Method, device, equipment and readable medium for generating relative posture information of camera device
By acquiring and processing object viewing image sets, generating three-dimensional point cloud information and eliminating invalid feature points, the problem of waste of computing power and high retrieval complexity caused by invalid feature points in the prior art is solved, and a more efficient camera equipment is achieved Pose information generation.
Patent Information
- Application Number
- CN202410051549.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-12
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-01-12
AI Technical Summary
When generating relative pose information of the camera device, the existence of invalid feature points participating in the matching process leads to wasting computer computing resources, and the complexity is high when retrieving similarity matching, resulting in a long time consumption.
By obtaining the object's viewing image set, the initial feature point information is extracted, the feature point matching prediction model is input to generate three-dimensional point cloud information, and invalid feature points are eliminated, and the common view feature point information is matched to generate relative pose information.
It reduces the waste of computer computing resources, reduces the complexity of retrieval time, and improves the efficiency of relative pose information generation.
Smart Images

Figure CN117808881B_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a method, apparatus, device, and readable medium for generating relative posture information of a camera device. Background Art
[0002] With the development of augmented reality (AR) technology, in order to make the virtual objects rendered by AR devices closely integrated with real objects to achieve a visual effect of virtual-real fusion, more accurate relative pose information between the objects in the images taken by the camera device and the camera device is needed. The relative pose information generation of the camera device is a technology for generating the relative pose information between the objects in the images taken by the camera device and the camera device. At present, when generating the relative pose information between the object and the camera device, the method usually adopted is: reconstructing each three-dimensional feature point of the object in the three-dimensional space through the feature point information of each perspective map, and directly generating the relative pose information based on the feature point information of each perspective map without removing the invalid feature points, the three-dimensional feature point information and the two-dimensional feature point information corresponding to the image taken by the camera device.
[0003] However, when the above method is used to generate relative pose information, the following technical problems often occur:
[0004] First, when reconstructing the three-dimensional feature points of an object in three-dimensional space through the feature point information of each view image, due to the self-occlusion of the feature points under different view angles and the limited view ranges corresponding to each view image, some feature points on each view image have no corresponding three-dimensional feature points after reconstructing each three-dimensional feature point. These feature points without corresponding three-dimensional feature points (i.e., invalid feature points) will still participate in the subsequent matching and relative pose information generation process when matching the correspondence between the three-dimensional feature point information and the two-dimensional feature point information corresponding to the image captured by the camera device through the feature point information of each feature point on each view image, resulting in a waste of computer computing resources.
[0005] However, it is found in practice that when searching for the common view feature point information group with the highest feature similarity to the above-mentioned feature point information set of the image of the object to be located, there are further technical problems as follows:
[0006] Second, directly matching each image feature point information of the object to be located in the image feature point information set of the object to be located with each common view feature point information in each common view feature point information group set one by one, determining the similarity of each two matched feature point information, and then determining the similarity of the image feature point information of the object to be located with each common view feature point information group in each common view feature point information group set. By retrieving the similarity between the image feature point information of the object to be located and each common view feature point information group in each common view feature point information group set, determining the common view feature point information group with the highest feature similarity to the image feature point information set of the object to be located in each common view feature point information group set increases the complexity of computer processing, so that the computer takes a long time to retrieve the common view feature point information group with the highest similarity to the image feature point information set of the object to be located.
[0007] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention
[0008] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.
[0009] Some embodiments of the present disclosure propose methods, devices, electronic devices and computer-readable media for generating relative posture information of a camera device to solve one or more of the technical problems mentioned in the above background technology section.
[0010] In a first aspect, some embodiments of the present disclosure provide a method for generating relative posture information of a camera device, the method comprising: obtaining an object perspective image set corresponding to a positioned object, wherein each object perspective image in the object perspective image set corresponds to each shooting perspective; for each object perspective image in the object perspective image set, performing feature point information extraction processing on the object perspective image to obtain an initial feature point information group corresponding to the positioned object; determining each obtained initial feature point information group as an initial feature point information group set, wherein an object perspective image in the object perspective image set corresponds to an initial feature point information group corresponding to the initial feature point information group set; inputting the initial feature point information group set into a pre-trained feature point matching prediction model to obtain each matching feature point prediction information group; based on each matching feature point prediction information group, generating a three-dimensional point cloud information set corresponding to each matching feature point prediction information group; for each initial feature point information group in the initial feature point information group set, performing invalid feature point information elimination processing on the initial feature point information group according to each matching feature point prediction information group to obtain a feature point information group; determining each obtained feature point information group as a feature point information A set, wherein a feature point information group in the feature point information set corresponds to a corresponding piece of three-dimensional point cloud information in the three-dimensional point cloud information set, and each feature point information in the feature point information set corresponds to the three-dimensional point cloud information; obtaining an image of the object to be located corresponding to the above-mentioned positioning object collected by a camera device; performing feature point information extraction processing on the image of the object to be located to generate a feature point information set of the image of the object to be located corresponding to the image of the object to be located; based on the feature point information set of the image of the object to be located, the feature point information set and the three-dimensional point cloud information set, generating each common view feature point information set; Based on the above-mentioned image feature point information set of the object to be located and the above-mentioned common view feature point information group sets, at least three image feature point information of the object to be located in the above-mentioned image feature point information set are used as respective two-dimensional feature point information to be matched with the three-dimensional point cloud information in the above-mentioned three-dimensional point cloud information set to obtain respective matching feature point information groups, wherein the matching feature point information groups in the above-mentioned respective matching feature point information groups include one image feature point information of the object to be located and one three-dimensional point cloud information; based on the above-mentioned respective matching feature point information groups, the relative posture information of the positioning object in the above-mentioned image of the object to be located and the above-mentioned camera device is generated.
[0011] In a second aspect, some embodiments of the present disclosure provide a device for generating relative posture information of a camera device, the device comprising: a first acquisition unit, configured to acquire an object perspective image set corresponding to a positioned object, wherein each object perspective image in the object perspective image set corresponds to each shooting perspective; a first extraction and processing unit, configured to perform feature point information extraction processing on each object perspective image in the object perspective image set, to obtain an initial feature point information group corresponding to the positioned object; a first determination unit, configured to determine each obtained initial feature point information group as an initial feature point information group set, wherein an object perspective image in the object perspective image set corresponds to the initial feature point information group. The first information group includes an initial feature point information group corresponding to an initial feature point information group in the information group set; an input unit is configured to input the above initial feature point information group set into a pre-trained feature point matching prediction model to obtain each matching feature point prediction information group; the first generating unit is configured to generate a three-dimensional point cloud information set corresponding to each matching feature point prediction information group based on the above matching feature point prediction information groups; the elimination processing unit is configured to, for each initial feature point information group in the above initial feature point information group set, perform invalid feature point information elimination processing on the above initial feature point information group according to the above matching feature point prediction information groups to obtain a feature point information group; the second determining unit is configured to obtain each feature point information group , determined as a feature point information set, wherein a feature point information set in the feature point information set corresponds to a three-dimensional point cloud information corresponding to the three-dimensional point cloud information set, and each feature point information in the feature point information set corresponds to the three-dimensional point cloud information; a second acquisition unit is configured to acquire an image of the object to be located corresponding to the above-mentioned positioning object acquired by the camera device; a second extraction processing unit is configured to perform feature point information extraction processing on the above-mentioned image of the object to be located, and generate a feature point information set of the image of the object to be located corresponding to the above-mentioned image of the object to be located; a second generation unit is configured to generate each feature point information set based on the feature point information set of the image of the object to be located, the feature point information set and the three-dimensional point cloud information set. A common view feature point information set; a matching unit, configured to match at least three image feature point information of the object to be located in the above-mentioned image feature point information set as each two-dimensional feature point information with the three-dimensional point cloud information in the above-mentioned three-dimensional point cloud information set based on the above-mentioned image feature point information set of the object to be located and the above-mentioned each common view feature point information set, to obtain each two-dimensional and three-dimensional matching feature point information group, wherein the matching feature point information group in the above-mentioned each matching feature point information group includes one image feature point information of the object to be located and one three-dimensional point cloud information; a third generating unit, configured to generate relative posture information of the positioning object in the above-mentioned image of the object to be located and the above-mentioned camera device based on the above-mentioned each matching feature point information group.
[0012] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.
[0013] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.
[0014] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the relative pose information generation method of the camera device of some embodiments of the present disclosure, the waste of computer computing power resources is reduced. Specifically, the reason for the waste of computer computing power resources is that when reconstructing the three-dimensional feature points of the object in the three-dimensional space through the feature point information of each perspective image, due to the self-occlusion of the feature points under different perspectives and the limited range of each perspective corresponding to each perspective image, some feature points on each perspective image have no corresponding three-dimensional feature points after reconstructing each three-dimensional feature point. These feature points without corresponding three-dimensional feature points (i.e., invalid feature points) will still participate in the subsequent matching and relative pose information generation process when matching the correspondence between the three-dimensional feature point information and the two-dimensional feature point information corresponding to the image captured by the camera device through the feature point information of each feature point on each perspective image, resulting in a waste of computer computing power resources. Based on this, the relative pose information generation method of the camera device of some embodiments of the present disclosure, first, obtains an object perspective image set corresponding to the positioning object, wherein each object perspective image in the above-mentioned object perspective image set corresponds to each shooting perspective. Thus, an object perspective image set for generating a three-dimensional point cloud information set can be obtained. Then, for each object perspective image in the above-mentioned object perspective image set, feature point information extraction processing is performed on the above-mentioned object perspective image to obtain the initial feature point information group corresponding to the above-mentioned positioning object. Thus, feature point information extraction processing can be performed on the object perspective image to obtain the initial feature point information group corresponding to the object perspective image. Then, each obtained initial feature point information group is determined as an initial feature point information group set, wherein an object perspective image in the above-mentioned object perspective image set corresponds to an initial feature point information group corresponding to the above-mentioned initial feature point information group set. Afterwards, the above-mentioned initial feature point information group set is input into a pre-trained feature point matching prediction model to obtain each matching feature point prediction information group. Thus, the initial feature point information in the initial feature point information group set can be matched by the feature point matching prediction model to obtain each matching feature point prediction information group. Then, based on the above-mentioned matching feature point prediction information groups, a three-dimensional point cloud information set corresponding to the above-mentioned matching feature point prediction information groups is generated. Thus, a three-dimensional point cloud information set corresponding to the three-dimensional model representing the positioning object in three-dimensional space can be obtained. Next, for each initial feature point information group in the initial feature point information group set, invalid feature point information is eliminated from the initial feature point information group according to each matching feature point prediction information group to obtain a feature point information group. In this way, invalid feature points in the initial feature point information group can be eliminated.Then, each feature point information group obtained is determined as a feature point information group set, wherein one feature point information group in the feature point information group set corresponds to one three-dimensional point cloud information corresponding to the three-dimensional point cloud information set, and each feature point information in the feature point information group corresponds to the three-dimensional point cloud information. Thus, a feature point information group set used to generate each common view feature point information group set can be obtained. Next, an image of the object to be located corresponding to the above-mentioned positioning object collected by the camera device is obtained. Thus, an image of the object to be located for determining the relative position information between the positioning object in the image and the camera device can be obtained. Then, feature point information extraction processing is performed on the above-mentioned image of the object to be located to generate a feature point information set of the image of the object to be located corresponding to the above-mentioned image of the object to be located. Thus, feature point information extraction processing can be performed on the image of the object to be located to obtain a feature point information set of the image of the object to be located. Afterwards, based on the feature point information set of the image of the object to be located, the feature point information group set and the three-dimensional point cloud information set, each common view feature point information group set is generated. Thus, each common view feature point information group set used to generate each matching feature point information group can be obtained. Then, based on the above-mentioned image feature point information set of the object to be located and the above-mentioned common view feature point information group sets, at least three image feature point information of the object to be located in the above-mentioned image feature point information set are used as each two-dimensional feature point information to match the three-dimensional point cloud information in the above-mentioned three-dimensional point cloud information set, and each matching feature point information group is obtained, wherein the matching feature point information group in the above-mentioned matching feature point information group includes one image feature point information of the object to be located and one three-dimensional point cloud information. Thus, the corresponding relationship between the three-dimensional feature point information in the three-dimensional point cloud information set and the two-dimensional feature point information of the image taken by the corresponding camera device can be matched with the feature point information group after each invalid feature point elimination process. Finally, based on the above-mentioned matching feature point information groups, the relative posture information of the positioning object in the above-mentioned image of the object to be located and the above-mentioned camera device is generated. Thus, the relative posture information of the positioning object in the image of the object to be located and the above-mentioned camera device can be generated. This is also because after generating the three-dimensional point cloud information set corresponding to each three-dimensional feature point of the object in the three-dimensional space according to each initial feature point information group corresponding to each object's viewpoint image, invalid feature point information is eliminated from each initial feature point information group to obtain each feature point information group after the invalid feature point elimination process is performed. By using each feature point information group after the invalid feature point elimination process as a medium, the correspondence between the three-dimensional feature point information in the three-dimensional point cloud information set and the two-dimensional feature point information of the image captured by the corresponding camera device is matched. This reduces the waste of computer computing resources in the process of matching and generating relative posture information. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.
[0016] Figure 1 is a flowchart of some embodiments of a method for generating relative posture information of a camera device according to the present disclosure;
[0017] Figure 2 It is a schematic structural diagram of some embodiments of the apparatus for generating relative posture information of a camera device according to the present disclosure;
[0018] Figure 3 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION
[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0020] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.
[0021] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0022] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0023] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0024] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.
[0025] Figure 1The process 100 of some embodiments of the method for generating relative posture information of a camera device according to the present disclosure is shown. The method for generating relative posture information of a camera device comprises the following steps:
[0026] Step 101: Acquire an object view image set corresponding to the positioned object.
[0027] In some embodiments, the execution subject (for example, a computing device) of the method for generating relative posture information of a camera device may obtain a set of object perspective images corresponding to the positioning object through a wired connection or a wireless connection. Among them, each object perspective image in the above object perspective image set corresponds to each shooting perspective. An object perspective image in the above object perspective image set corresponds to one of the above shooting perspectives. In practice, the above execution subject may obtain a set of object perspective images corresponding to the positioning object from a preset database. Among them, the above preset database may be a MySQL database. The object perspective images in the above object perspective image set may be images containing the positioning object taken from a preset angle. The above execution subject may be a computing device for generating relative posture information between an object in an image taken by a camera device and a camera device. The above camera device may be a device on an AR device. For example, the above camera device may be a camera on an AR head-mounted display device.
[0028] It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.
[0029] Step 102 : For each object perspective image in the object perspective image set, feature point information extraction processing is performed on the object perspective image to obtain an initial feature point information group corresponding to the positioned object.
[0030] In some embodiments, the execution entity may perform feature point information extraction processing on each object perspective image in the object perspective image set to obtain an initial feature point information group corresponding to the positioning object.
[0031] In some optional implementations of some embodiments, the execution subject may perform feature point information extraction processing on the object view image through the following steps to obtain an initial feature point information group corresponding to the positioning object:
[0032] The first step is to perform background removal processing on the object perspective image to obtain a background-removed object perspective image. In practice, the execution subject can input the object perspective image into a pre-trained background removal model to obtain a background-removed object perspective image. The background removal model can be a Gaussian Mixture Model (GMM).
[0033] In the second step, the background-removed object view image is input into a pre-trained feature point extraction model to obtain an initial feature point information group corresponding to the positioning object. The feature point extraction model may be a feature extraction network HS-ResNet model. The initial feature point information in the initial feature point information group may represent information corresponding to the feature points in the background-removed object view image. The initial feature point information includes but is not limited to at least one of the following: two-dimensional coordinates, intensity information. The intensity information may represent the grayscale value of the feature point.
[0034] Step 103: determine the obtained initial feature point information groups as an initial feature point information group set.
[0035] In some embodiments, the execution subject may determine the obtained initial feature point information groups as an initial feature point information group set, wherein one object perspective image in the object perspective image set corresponds to one initial feature point information group corresponding to the initial feature point information group set.
[0036] Step 104: input the initial feature point information group set into a pre-trained feature point matching prediction model to obtain each matching feature point prediction information group.
[0037] In some embodiments, the execution subject may input the initial feature point information group set into a pre-trained feature point matching prediction model to obtain each matching feature point prediction information group. The matching feature point prediction information in the matching feature point prediction information group may represent each initial feature point information corresponding to a point in three-dimensional space under different viewing angles. The feature point matching prediction model includes a feature vector generation layer, an attention mechanism layer, a multi-layer perceptron layer, and an output layer.
[0038] In some optional implementations of some embodiments, the execution subject may input the initial feature point information group set into a pre-trained feature point matching prediction model through the following steps to obtain each matching feature point prediction information group:
[0039] In the first step, the initial feature point information set is input into the feature vector generation layer to obtain a feature vector set. One initial feature point information set in the initial feature point information set corresponds to one feature vector set corresponding to the feature vector set in the feature vector set. The feature vector generation layer may be a pooling layer.
[0040] In the second step, the feature vector set is input into the attention mechanism layer, and the correlation weight information between each two feature vectors that are not in the same group in the feature vector set is generated to obtain each correlation weight information. Among them, the attention mechanism layer can be a self-attention layer. The correlation weight information can represent the weight of the similarity between two feature vectors. The correlation weight information can include a weight value and each feature vector corresponding to the weight value. The similarity can be a cosine similarity.
[0041] In the third step, the above-mentioned correlation weight information is input into the above-mentioned multi-layer perceptron layer to obtain each matching feature point prediction information group information. Wherein, the above-mentioned multi-layer perceptron layer includes each fully connected layer. The matching feature point prediction information group information in the above-mentioned matching feature point prediction information group information can represent each initial feature point information under each different perspective corresponding to a point in the three-dimensional space and the confidence information of the corresponding predicted matching feature point prediction information group.
[0042] The fourth step is to input the information of each matching feature point prediction information group into the output layer to obtain each matching feature point prediction information group.
[0043] Step 105: Based on each matching feature point prediction information group, a three-dimensional point cloud information set corresponding to each matching feature point prediction information group is generated.
[0044] In some embodiments, the execution subject may generate a three-dimensional point cloud information set corresponding to each of the matching feature point prediction information groups based on the matching feature point prediction information groups. In practice, the execution subject may perform motion estimation on each of the matching feature point prediction information groups through the SFM (Structure-from-Motion) algorithm to obtain camera extrinsics. Then, the execution subject may perform three-dimensional structure recovery and reconstruction based on each of the matching feature point prediction information groups and the camera extrinsics through the SFM (Structure-from-Motion) algorithm to obtain a three-dimensional point cloud information set for each of the matching feature point prediction information groups. Among them, the three-dimensional point cloud information in the three-dimensional point cloud information set may represent a three-dimensional point in the three-dimensional space corresponding to the positioning object. The three-dimensional point cloud information may include, but is not limited to, at least one of the following: three-dimensional coordinates, RGB color values, and normal vectors.
[0045] Step 106 , for each initial feature point information group in the initial feature point information group set, according to each matching feature point prediction information group, perform invalid feature point information elimination processing on the initial feature point information group to obtain a feature point information group.
[0046] In some embodiments, the execution subject may remove invalid feature point information from each initial feature point information group in the initial feature point information group set according to each matching feature point prediction information group to obtain a feature point information group. The invalid feature point information may be initial feature point information in the initial feature point information group set that has no corresponding relationship with the three-dimensional point cloud information in the three-dimensional point cloud information set.
[0047] In some optional implementations of some embodiments, the execution subject may perform the following steps to remove invalid feature point information from the initial feature point information group according to the matching feature point prediction information groups to obtain a feature point information group:
[0048] In the first step, each matching feature point prediction information included in each matching feature point prediction information group in the above-mentioned matching feature point prediction information groups is determined as each target matching feature point prediction information.
[0049] In the second step, the determined prediction information of each target matching feature point is determined as a target matching feature point prediction information set.
[0050] In the third step, in response to determining that there is initial feature point information in the above-mentioned initial feature point information group that is different from the target matching feature point prediction information in the above-mentioned target matching feature point prediction information set, each initial feature point information different from the target matching feature point prediction information is deleted from the above-mentioned initial feature point information group to update the above-mentioned initial feature point information group.
[0051] In the fourth step, the updated initial feature point information group is determined as the feature point information group.
[0052] Step 107: determine the obtained feature point information groups as a feature point information group set.
[0053] In some embodiments, the execution subject may determine the obtained feature point information groups as a feature point information group set, wherein one feature point information group in the feature point information group set corresponds to one corresponding three-dimensional point cloud information in the three-dimensional point cloud information set, and each feature point information in the feature point information group corresponds to the three-dimensional point cloud information.
[0054] Step 108: Acquire the image of the object to be positioned corresponding to the positioning object captured by the camera device.
[0055] In some embodiments, the execution subject may obtain an image of the object to be located corresponding to the positioning object captured by a camera. The camera includes but is not limited to one of the following: a camera, a camera head. The image of the object to be located may be an image containing the positioning object. The shooting time of the image of the object to be located is after the shooting time of each object perspective image in the object perspective image set. The shooting angle of the image of the object to be located may be a random shooting angle.
[0056] Step 109 , extracting feature point information from the image of the object to be located, and generating a feature point information set of the image of the object to be located corresponding to the image of the object to be located.
[0057] In some embodiments, the execution subject may perform feature point information extraction processing on the image of the object to be located, and generate a feature point information set of the image of the object to be located corresponding to the image of the object to be located. In practice, the execution subject may input the image of the object to be located into the pre-trained feature point extraction model, and obtain a feature point information set of the image of the object to be located corresponding to the image of the object to be located. Among them, the feature point information of the image of the object to be located in the feature point information set of the image of the object to be located may represent information corresponding to feature points extracted from the image of the object to be located. The feature point information of the image of the object to be located includes but is not limited to at least one of the following: two-dimensional coordinates, intensity information. The feature point information of the image of the object to be located may be represented by a vector.
[0058] Step 110 , generating each common view feature point information set based on the image feature point information set of the object to be located, the feature point information set and the three-dimensional point cloud information set.
[0059] In some embodiments, the execution entity may generate each common view feature point information set based on the image feature point information set of the object to be located, the feature point information set and the three-dimensional point cloud information set.
[0060] In some optional implementations of some embodiments, the execution subject may generate each common view feature point information set based on the image feature point information set of the object to be located, the feature point information set and the three-dimensional point cloud information set through the following steps:
[0061] In the first step, the number of feature point information included in each feature point information group in the feature point information group set is determined as a sorting quantity to obtain various sorting quantities, wherein one sorting quantity in the various sorting quantities corresponds to a corresponding feature point information group in the feature point information group set.
[0062] The second step is to store each feature point information group in the feature point information group set into a preset candidate queue according to the above-mentioned sorting quantities. The above-mentioned preset candidate queue can be a preset queue. In practice, the above-mentioned execution subject can store the above-mentioned feature point information groups into the above-mentioned preset candidate queue according to the sorting quantities corresponding to the above-mentioned feature point information groups from large to small according to the sorting quantities.
[0063] The third step is to perform the following aggregation processing on each feature point information group in the preset candidate queue:
[0064] Sub-step one: for the first feature point information group in the preset candidate queue, each three-dimensional point cloud information corresponding to each feature point information in the first feature point information group in the above three-dimensional point cloud information set is determined as the first target three-dimensional point cloud information set.
[0065] Sub-step 2: determining the first feature point information group as the common view feature point information group.
[0066] Sub-step three: determining each feature point information group except the first feature point information group in the preset candidate queue as a target feature point information group set.
[0067] Sub-step 4: for each target feature point information group in the target feature point information group set, perform the following steps:
[0068] In the first sub-step, each piece of three-dimensional point cloud information in the three-dimensional point cloud information set corresponding to each piece of target feature point information in the target feature point information group is determined as a second target three-dimensional point cloud information set.
[0069] The second sub-step is to determine the target feature point information group as a common view feature point information group in response to determining that the amount of three-dimensional point cloud information in the intersection of the first target three-dimensional point cloud information set and the second target three-dimensional point cloud information set is greater than a preset threshold.
[0070] Sub-step five: determining each determined common view feature point information group as a common view feature point information group set.
[0071] Sub-step six: deleting the first feature point information group from the preset candidate queue to update the preset candidate queue.
[0072] Sub-step seven, in response to determining that the preset candidate queue is not empty, performing the above aggregation process again according to the updated preset candidate queue.
[0073] Step 111, based on the image feature point information set of the object to be located and each common view feature point information group set, at least three image feature point information of the object to be located in the image feature point information set of the object to be located are matched with the three-dimensional point cloud information in the three-dimensional point cloud information set as each two-dimensional feature point information to obtain each matching feature point information group.
[0074] In some embodiments, the execution subject may match at least three pieces of image feature point information of the object to be located in the image feature point information set of the object to be located as two-dimensional feature point information with three-dimensional point cloud information in the three-dimensional point cloud information set based on the image feature point information set of the object to be located and each of the common view feature point information groups to obtain each matching feature point information group. The matching feature point information group in each of the matching feature point information groups includes one piece of image feature point information of the object to be located and one piece of three-dimensional point cloud information.
[0075] In some optional implementations of some embodiments, the execution subject may perform the following steps based on the image feature point information set of the object to be located and the respective common view feature point information sets, and match at least three image feature point information of the object to be located in the image feature point information set of the object to be located as respective two-dimensional feature point information with the three-dimensional point cloud information in the three-dimensional point cloud information set to obtain respective matching feature point information sets:
[0076] In the first step, based on the above-mentioned image feature point information set of the object to be located and the above-mentioned common view feature point information group sets, the common view feature point information group with the highest feature similarity with the above-mentioned image feature point information set of the object to be located is retrieved as the target common view feature point information group. The above-mentioned feature similarity can be cosine similarity.
[0077] In the second step, based on the above-mentioned image feature point information set of the object to be located, the above-mentioned target common view feature point information set and the above-mentioned three-dimensional point cloud information set, each matching feature point information set is generated.
[0078] In some optional implementations of some embodiments, the execution subject may retrieve the common view feature point information group with the highest feature similarity with the image feature point information set of the object to be located from among the common view feature point information group sets as the target common view feature point information group based on the image feature point information set of the object to be located and the common view feature point information group sets through the following steps:
[0079] The first step is to perform feature aggregation processing on each of the image feature point information of the object to be located in the above-mentioned image feature point information set to obtain an aggregated feature vector. In practice, the above-mentioned execution subject can aggregate the above-mentioned image feature point information of each of the object to be located into a feature vector as the aggregated feature vector through feature fusion technology.
[0080] In the second step, for each common view feature point information group set in the above common view feature point information group sets, the following steps are performed:
[0081] Sub-step 1: for each common view feature point information group in the common view feature point information group set, feature aggregation processing is performed on each common view feature point information in the common view feature point information group to generate an aggregated common view feature vector corresponding to the common view feature point information group. In practice, the execution subject may aggregate each common view feature point information into a feature vector as an aggregated common view feature vector through feature fusion technology.
[0082] Sub-step 2: performing feature fusion processing on each generated aggregated common view feature vector to obtain an indexed aggregated common view feature vector corresponding to the above common view feature point information group set. The above indexed aggregated common view feature vector may be a vector obtained by performing feature fusion processing on each of the above aggregated common view feature vectors.
[0083] In the third step, the obtained indexed aggregated common view feature vectors are determined as an indexed aggregated common view feature vector set, wherein one of the indexed aggregated common view feature vectors corresponds to one of the common view feature point information group sets in the common view feature point information group sets.
[0084] In the fourth step, the similarity between the aggregated feature vector and each indexed aggregated common view feature vector in the indexed aggregated common view feature vector set is determined as the target similarity to obtain a target similarity set. The similarity may be cosine similarity.
[0085] In the fifth step, the index-aggregated common-view feature vector corresponding to the maximum target similarity in the target similarity set is determined as the target index-aggregated common-view feature vector.
[0086] In the sixth step, the common view feature point information group set corresponding to the target index aggregated common view feature vector is determined as the target common view feature point information group set.
[0087] In the seventh step, the aggregated common view feature vector corresponding to each target common view feature point information group in the above target common view feature point information group set is determined as the target aggregated common view feature vector to obtain each target aggregated common view feature vector.
[0088] In the eighth step, the similarity between each target aggregated common view feature vector in the above target aggregated common view feature vectors and the above aggregated feature vector is determined as the similarity to be screened, so as to obtain each similarity to be screened. The above similarity may be a cosine similarity.
[0089] In the ninth step, the target aggregated common-view feature vector corresponding to the largest similarity to be screened among the above similarities to be screened is determined as the target aggregated common-view feature vector to be queried.
[0090] In the tenth step, the common view feature point information group corresponding to the aggregated common view feature vector of the target to be queried is determined as the target common view feature point information group.
[0091] The above-mentioned technical scheme and its related contents, as an inventive point of an embodiment of the present disclosure, solve the second technical problem mentioned in the background technology, which is "directly matching each image feature point information of the object to be located in the image feature point information set of the object to be located with each common view feature point information in each common view feature point information group set one by one, determining the similarity of each two matched feature point information, and then determining the similarity between the image feature point information of the object to be located and each common view feature point information group in each common view feature point information group set. By retrieving the similarity between the image feature point information of the object to be located and each common view feature point information group in each common view feature point information group set, the common view feature point information group with the highest feature similarity to the image feature point information set of the object to be located in each common view feature point information group set is determined, which increases the complexity of computer processing, so that the computer takes a long time to retrieve the common view feature point information group with the highest similarity to the image feature point information set of the object to be located." The factors that cause the computer to take a long time to retrieve are often as follows: directly matching each image feature point information of the object to be located in the image feature point information set of the object to be located with each common view feature point information in each common view feature point information group set, determining the similarity of each two matched feature point information, and then determining the similarity of the image feature point information of the object to be located with each common view feature point information group in each common view feature point information group set. By retrieving the similarity between the image feature point information of the object to be located and each common view feature point information group in each common view feature point information group set, the common view feature point information group with the highest feature similarity to the image feature point information set of the object to be located in each common view feature point information group set is determined, which increases the complexity of computer processing, making the computer take a long time to retrieve the common view feature point information group with the highest similarity to the image feature point information set of the object to be located. If the above factors are solved, the retrieval time of the computer when retrieving the common view feature point information group with the highest similarity to the image feature point information set of the object to be located can be reduced. In order to achieve this effect, first, feature aggregation processing is performed on each image feature point information of the object to be located in the above-mentioned image feature point information set of the object to be located to obtain an aggregated feature vector. In this way, an aggregated feature vector for retrieval can be obtained. Then, for each common view feature point information group set in the above-mentioned common view feature point information group set, the following steps are performed: Sub-step one, for each common view feature point information group in the above-mentioned common view feature point information group set, feature aggregation processing is performed on each common view feature point information in the above-mentioned common view feature point information group to generate an aggregated common view feature vector corresponding to the above-mentioned common view feature point information group. In this way, each aggregated common view feature vector representing each common view feature point information group in the common view feature point information group set can be obtained. Sub-step two, feature fusion processing is performed on each generated aggregated common view feature vector to obtain an indexed aggregated common view feature vector corresponding to the above-mentioned common view feature point information group set.Thus, an indexed aggregated common view feature vector representing a set of common view feature point information groups can be obtained. Then, each obtained indexed aggregated common view feature vector is determined as an indexed aggregated common view feature vector set, wherein one indexed aggregated common view feature vector in each of the above indexed aggregated common view feature vectors corresponds to a common view feature point information group set corresponding to each of the above common view feature point information group sets. Thus, an indexed aggregated common view feature vector set for retrieval can be obtained. Then, the similarity between the above aggregated feature vector and each indexed aggregated common view feature vector in the above indexed aggregated common view feature vector set is determined as a target similarity to obtain a target similarity set. Thus, a target similarity set for determining a set of common view feature point information groups in which the common view feature point information group most similar to the above image feature point information set of the object to be located is located can be obtained. Then, the indexed aggregated common view feature vector corresponding to the maximum target similarity in the target similarity set is determined as a target indexed aggregated common view feature vector. Thus, a target indexed aggregated common view feature vector corresponding to a set of common view feature point information groups in which the common view feature point information group most similar to the above image feature point information set of the object to be located is located can be obtained. Next, the set of common view feature point information groups corresponding to the above target index aggregate common view feature vector is determined as the target common view feature point information group set. Thus, the set of common view feature point information groups containing the common view feature point information group that is most similar to the above image feature point information set of the object to be located can be obtained. Then, the aggregate common view feature vector corresponding to each target common view feature point information group in the above target common view feature point information group set is determined as the target aggregate common view feature vector, and each target aggregate common view feature vector is obtained. Thus, each target aggregate common view feature vector used to retrieve the common view feature point information group that is most similar to the image feature point information set of the object to be located in the target common view feature point information group set can be obtained. Then, the similarity between each target aggregate common view feature vector in the above target aggregate common view feature vectors and the above aggregate feature vector is determined as the similarity to be screened, and each similarity to be screened is obtained. Thus, each similarity to be screened for retrieval can be obtained. The target aggregate common view feature vector corresponding to the largest similarity to be screened among the above similarities to be screened is determined as the target aggregate common view feature vector to be queried. Thus, the aggregated common view feature vector of the target to be queried corresponding to the common view feature point information group that is most similar to the image feature point information set of the object to be located can be obtained. Finally, the common view feature point information group corresponding to the above-mentioned aggregated common view feature vector of the target to be queried is determined as the target common view feature point information group. Thus, the common view feature point information group with the highest feature similarity to the above-mentioned image feature point information set of the object to be located can be retrieved. Also because of the generation of aggregated feature vectors for retrieval, the indexed aggregated common view feature vector set, and each aggregated common view feature vector.The target common view feature point information group set is determined by aggregating feature vectors and indexing the aggregated common view feature vector set to include the common view feature point information group that is most similar to the image feature point information set of the object to be located. Then, the common view feature point information group that is most similar to the image feature point information set of the object to be located is retrieved from the target common view feature point information group set by using the aggregated feature vectors and each aggregated common view feature vector corresponding to the target common view feature point information group set. This reduces the complexity of computer retrieval and reduces the retrieval time of the computer retrieval of the common view feature point information group that is most similar to the image feature point information set of the object to be located.
[0092] In some optional implementations of some embodiments, the execution subject may generate each matching feature point information group based on the image feature point information set of the object to be located, the target common view feature point information group and the three-dimensional point cloud information set through the following steps:
[0093] The first step is to input the above-mentioned image feature point information set of the object to be located and the above-mentioned target common view feature point information group into a pre-trained image feature point matching model to obtain each initial matching feature point information group. Among them, each initial matching feature point information group in the above-mentioned initial matching feature point information groups includes an image feature point information of the object to be located and a feature point information. The above-mentioned image feature point matching model can be a feature matching network SuperGlue model. An initial matching feature point information in each of the above-mentioned initial matching feature point information groups can represent the correspondence between an image feature point information of the object to be located in the image feature point information set of the object to be located and a corresponding feature point information in the above-mentioned target common view feature point information group.
[0094] In the second step, for each of the above-mentioned initial matching feature point information groups, the following steps are performed:
[0095] Sub-step 1: determining the feature point information included in the initial matching feature point information group as the feature point information to be replaced.
[0096] Sub-step two: determining the three-dimensional point cloud information corresponding to the feature point information to be replaced as the replacement point cloud information.
[0097] Sub-step three: updating the feature point information included in the above-mentioned initial matching feature point information group to the replacement point cloud information, so as to update the initial matching feature point information group.
[0098] Sub-step four: determining the updated initial matching feature point information group as the matching feature point information group.
[0099] Step 112: Generate relative position information of the positioning object and the camera device in the image of the object to be positioned based on each matching feature point information group.
[0100] In some embodiments, the execution subject may generate the relative posture information of the positioning object in the image of the object to be positioned and the camera device based on the matching feature point information groups. In practice, for each matching feature point information group in each matching feature point information group, first, the execution subject may determine the two-dimensional coordinates in the feature point information of the image of the object to be positioned included in the matching feature point information group as the target two-dimensional coordinates. Then, the execution subject may determine the three-dimensional coordinates in the three-dimensional point cloud information included in the matching feature point information group as the target three-dimensional coordinates. After that, the execution subject may determine the target two-dimensional coordinates and the target three-dimensional coordinates as a coordinate group. Next, the execution subject may use the determined coordinate groups as the input of the PnP algorithm. Then, the execution subject may determine the rotation matrix and translation vector output by the PnP algorithm as the relative posture information between the camera device and the positioning object.
[0101] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the relative pose information generation method of the camera device of some embodiments of the present disclosure, the waste of computer computing power resources is reduced. Specifically, the reason for the waste of computer computing power resources is that when reconstructing the three-dimensional feature points of the object in the three-dimensional space through the feature point information of each perspective image, due to the self-occlusion of the feature points under different perspectives and the limited range of each perspective corresponding to each perspective image, some feature points on each perspective image have no corresponding three-dimensional feature points after reconstructing each three-dimensional feature point. These feature points without corresponding three-dimensional feature points (i.e., invalid feature points) will still participate in the subsequent matching and relative pose information generation process when matching the correspondence between the three-dimensional feature point information and the two-dimensional feature point information corresponding to the image captured by the camera device through the feature point information of each feature point on each perspective image, resulting in a waste of computer computing power resources. Based on this, the relative pose information generation method of the camera device of some embodiments of the present disclosure, first, obtains an object perspective image set corresponding to the positioning object, wherein each object perspective image in the above-mentioned object perspective image set corresponds to each shooting perspective. Thus, an object perspective image set for generating a three-dimensional point cloud information set can be obtained. Then, for each object perspective image in the above-mentioned object perspective image set, feature point information extraction processing is performed on the above-mentioned object perspective image to obtain the initial feature point information group corresponding to the above-mentioned positioning object. Thus, feature point information extraction processing can be performed on the object perspective image to obtain the initial feature point information group corresponding to the object perspective image. Then, each obtained initial feature point information group is determined as an initial feature point information group set, wherein an object perspective image in the above-mentioned object perspective image set corresponds to an initial feature point information group corresponding to the above-mentioned initial feature point information group set. Afterwards, the above-mentioned initial feature point information group set is input into a pre-trained feature point matching prediction model to obtain each matching feature point prediction information group. Thus, the initial feature point information in the initial feature point information group set can be matched by the feature point matching prediction model to obtain each matching feature point prediction information group. Then, based on the above-mentioned matching feature point prediction information groups, a three-dimensional point cloud information set corresponding to the above-mentioned matching feature point prediction information groups is generated. Thus, a three-dimensional point cloud information set corresponding to the three-dimensional model representing the positioning object in three-dimensional space can be obtained. Next, for each initial feature point information group in the initial feature point information group set, invalid feature point information is eliminated from the initial feature point information group according to each matching feature point prediction information group to obtain a feature point information group. In this way, invalid feature points in the initial feature point information group can be eliminated.Then, each feature point information group obtained is determined as a feature point information group set, wherein one feature point information group in the feature point information group set corresponds to one three-dimensional point cloud information corresponding to the three-dimensional point cloud information set, and each feature point information in the feature point information group corresponds to the three-dimensional point cloud information. Thus, a feature point information group set used to generate each common view feature point information group set can be obtained. Next, an image of the object to be located corresponding to the above-mentioned positioning object collected by the camera device is obtained. Thus, an image of the object to be located for determining the relative position information between the positioning object in the image and the camera device can be obtained. Then, feature point information extraction processing is performed on the above-mentioned image of the object to be located to generate a feature point information set of the image of the object to be located corresponding to the above-mentioned image of the object to be located. Thus, feature point information extraction processing can be performed on the image of the object to be located to obtain a feature point information set of the image of the object to be located. Afterwards, based on the feature point information set of the image of the object to be located, the feature point information group set and the three-dimensional point cloud information set, each common view feature point information group set is generated. Thus, each common view feature point information group set used to generate each matching feature point information group can be obtained. Then, based on the above-mentioned image feature point information set of the object to be located and the above-mentioned common view feature point information group sets, at least three image feature point information of the object to be located in the above-mentioned image feature point information set are used as each two-dimensional feature point information to match the three-dimensional point cloud information in the above-mentioned three-dimensional point cloud information set, and each matching feature point information group is obtained, wherein the matching feature point information group in the above-mentioned matching feature point information group includes one image feature point information of the object to be located and one three-dimensional point cloud information. Thus, the corresponding relationship between the three-dimensional feature point information in the three-dimensional point cloud information set and the two-dimensional feature point information of the image taken by the corresponding camera device can be matched with the feature point information group after each invalid feature point elimination process. Finally, based on the above-mentioned matching feature point information groups, the relative posture information of the positioning object in the above-mentioned image of the object to be located and the above-mentioned camera device is generated. Thus, the relative posture information of the positioning object in the image of the object to be located and the above-mentioned camera device can be generated. This is also because after generating the three-dimensional point cloud information set corresponding to each three-dimensional feature point of the object in the three-dimensional space according to each initial feature point information group corresponding to each object's viewpoint image, invalid feature point information is eliminated from each initial feature point information group to obtain each feature point information group after the invalid feature point elimination process is performed. By using each feature point information group after the invalid feature point elimination process as a medium, the correspondence between the three-dimensional feature point information in the three-dimensional point cloud information set and the two-dimensional feature point information of the image captured by the corresponding camera device is matched. This reduces the waste of computer computing resources in the process of matching and generating relative posture information.
[0102] Further references Figure 2As an implementation of the methods shown in the figures, the present disclosure provides some embodiments of a device for generating relative posture information of a camera device, and these device embodiments are Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.
[0103] like Figure 2As shown, the relative posture information generating device 200 of the camera device of some embodiments includes: a first acquisition unit 201, a first extraction processing unit 202, a first determination unit 203, an input unit 204, a first generation unit 205, a elimination processing unit 206, a second determination unit 207, a second acquisition unit 208, a second extraction processing unit 209, a second generation unit 210, a matching unit 211 and a third generation unit 212.The first acquisition unit 201 is configured to acquire an object perspective image set corresponding to the positioned object, wherein each object perspective image in the object perspective image set corresponds to each shooting perspective; the first extraction processing unit 202 is configured to perform feature point information extraction processing on each object perspective image in the object perspective image set to obtain an initial feature point information group corresponding to the positioned object; the first determination unit 203 is configured to determine each obtained initial feature point information group as an initial feature point information group set, wherein an object perspective image in the object perspective image set corresponds to an initial feature point information group corresponding to the initial feature point information group set; the input unit 20 4 is configured to input the above-mentioned initial feature point information group set into a pre-trained feature point matching prediction model to obtain each matching feature point prediction information group; the first generating unit 205 is configured to generate a three-dimensional point cloud information set corresponding to each matching feature point prediction information group based on the above-mentioned each matching feature point prediction information group; the elimination processing unit 206 is configured to, for each initial feature point information group in the above-mentioned initial feature point information group set, perform invalid feature point information elimination processing on the above-mentioned initial feature point information group according to the above-mentioned each matching feature point prediction information group to obtain a feature point information group; the second determining unit 207 is configured to determine each obtained feature point information group as a feature point information group set, wherein the above A feature point information group in the feature point information group set corresponds to a three-dimensional point cloud information corresponding to the three-dimensional point cloud information set, and each feature point information in the feature point information group corresponds to the three-dimensional point cloud information; the second acquisition unit 208 is configured to acquire the image of the object to be located corresponding to the positioning object collected by the camera device; the second extraction processing unit 209 is configured to perform feature point information extraction processing on the image of the object to be located, and generate a feature point information set of the image of the object to be located corresponding to the image of the object to be located; the second generation unit 210 is configured to generate each common view feature point information based on the feature point information set of the image of the object to be located, the feature point information group set and the three-dimensional point cloud information set. The matching unit 211 is configured to match at least three image feature point information of the object to be located in the above-mentioned image feature point information set as each two-dimensional feature point information with the three-dimensional point cloud information in the above-mentioned three-dimensional point cloud information set based on the above-mentioned image feature point information set of the object to be located and the above-mentioned each common view feature point information set, so as to obtain each two-dimensional and three-dimensional matching feature point information group, wherein the matching feature point information group in the above-mentioned each matching feature point information group includes one image feature point information of the object to be located and one three-dimensional point cloud information; the third generating unit 212 is configured to generate the relative posture information of the positioning object in the above-mentioned image of the object to be located and the above-mentioned camera device based on the above-mentioned each matching feature point information group.
[0104] It is understood that the units described in the device 200 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 200 and the units contained therein, and will not be described in detail here.
[0105] Reference below Figure 3 , which shows a structural schematic diagram of an electronic device 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0106] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.
[0107] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.
[0108] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the functions defined in the method of some embodiments of the present disclosure are executed.
[0109] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.
[0110] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.
[0111] The computer-readable medium may be included in the electronic device; or may exist separately without being incorporated into the electronic device. A computer-readable medium carries one or more programs, and when one or more programs are executed by the electronic device, the electronic device: obtains an object perspective image set corresponding to the positioned object, wherein each object perspective image in the object perspective image set corresponds to each shooting perspective; for each object perspective image in the object perspective image set, performs feature point information extraction processing on the object perspective image to obtain an initial feature point information group corresponding to the positioned object; determines each obtained initial feature point information group as an initial feature point information group set, wherein an object perspective image in the object perspective image set corresponds to an initial feature point information group corresponding to the initial feature point information group set; inputs the initial feature point information group set into a pre-trained feature point matching prediction model to obtain each matching feature point prediction information group; based on each matching feature point prediction information group, generates a three-dimensional point cloud information set corresponding to each matching feature point prediction information group; for each initial feature point information group in the initial feature point information group set, performs invalid feature point information elimination processing on the initial feature point information group according to each matching feature point prediction information group to obtain a feature point information group; determines each obtained feature point information group as a feature point information set, wherein one feature point information set in the feature point information set corresponds to one piece of three-dimensional point cloud information corresponding to the three-dimensional point cloud information set, and each feature point information in the feature point information set corresponds to the three-dimensional point cloud information; obtaining an image of the object to be located corresponding to the above-mentioned object to be located collected by a camera device; performing feature point information extraction processing on the image of the object to be located to generate a feature point information set of the image of the object to be located corresponding to the image of the object to be located; generating each common view feature point information set based on the feature point information set of the image of the object to be located, the feature point information set and the three-dimensional point cloud information set ; Based on the above-mentioned image feature point information set of the object to be located and the above-mentioned common view feature point information groups, at least three image feature point information of the object to be located in the above-mentioned image feature point information set are used as respective two-dimensional feature point information to match with the three-dimensional point cloud information in the above-mentioned three-dimensional point cloud information set to obtain respective matching feature point information groups, wherein the matching feature point information groups in the above-mentioned each matching feature point information group include one image feature point information of the object to be located and one three-dimensional point cloud information; based on the above-mentioned each matching feature point information group, the relative posture information of the positioning object in the above-mentioned image of the object to be located and the above-mentioned camera device is generated.
[0112] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0113] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0114] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be set in a processor, for example, may be described as: a processor including a first acquisition unit, a first extraction processing unit, a first determination unit, an input unit 204, a first generation unit, a elimination processing unit, a second determination unit, a second acquisition unit, a second extraction processing unit, a second generation unit, a matching unit and a third generation unit. Among them, the names of these units do not constitute a limitation on the units themselves under certain circumstances. For example, the third generation unit may also be described as "a unit that generates relative posture information of the positioning object in the image of the object to be positioned and the above-mentioned camera device based on the above-mentioned matching feature point information groups".
[0115] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0116] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of technical features, but should also cover other technical solutions formed by any combination of technical features or their equivalent features without departing from the inventive concept. For example, a technical solution formed by replacing the features with (but not limited to) the technical features with similar functions disclosed in the embodiments of the present disclosure.
Claims
1. A method for generating relative posture information of a camera device, comprising: Acquire an object perspective image set corresponding to the positioned object, wherein each object perspective image in the object perspective image set corresponds to each shooting perspective; For each object perspective image in the object perspective image set, extracting feature point information from the object perspective image to obtain an initial feature point information group corresponding to the positioned object; Determining each obtained initial feature point information group as an initial feature point information group set, wherein one object perspective image in the object perspective image set corresponds to a corresponding initial feature point information group in the initial feature point information group set; Inputting the initial feature point information group set into a pre-trained feature point matching prediction model to obtain each matching feature point prediction information group; Based on the respective matching feature point prediction information groups, generating three-dimensional point cloud information sets corresponding to the respective matching feature point prediction information groups; For each initial feature point information group in the initial feature point information group set, according to each matching feature point prediction information group, the initial feature point information group is subjected to invalid feature point information elimination processing to obtain a feature point information group, wherein according to each matching feature point prediction information group, the initial feature point information group is subjected to invalid feature point information elimination processing to obtain a feature point information group, including: determining each matching feature point prediction information included in each matching feature point prediction information group in each matching feature point prediction information group as each target matching feature point prediction information; Determining each determined target matching feature point prediction information as a target matching feature point prediction information set; In response to determining that there is initial feature point information different from the target matching feature point prediction information in the target matching feature point prediction information set in the initial feature point information group, deleting each initial feature point information different from the target matching feature point prediction information from the initial feature point information group to update the initial feature point information group; Determine the updated initial feature point information group as the feature point information group; Determine the obtained feature point information groups as a feature point information group set, wherein one feature point information group in the feature point information group set corresponds to one corresponding piece of three-dimensional point cloud information in the three-dimensional point cloud information set, and each feature point information in the feature point information group corresponds to the three-dimensional point cloud information; Acquire an image of the object to be positioned corresponding to the positioning object captured by a camera device; Performing feature point information extraction processing on the image of the object to be located to generate a feature point information set of the image of the object to be located corresponding to the image of the object to be located; Based on the image feature point information set of the object to be located, the feature point information set and the three-dimensional point cloud information set, each common view feature point information set is generated, wherein the generation of each common view feature point information set based on the image feature point information set of the object to be located, the feature point information set and the three-dimensional point cloud information set includes: Determine the number of feature point information included in each feature point information group in the feature point information group set as a sorting quantity, and obtain respective sorting quantities, wherein one sorting quantity in the respective sorting quantities corresponds to a corresponding feature point information group in the feature point information group set; According to the respective sorting quantities, storing each feature point information group in the feature point information group set into a preset candidate queue; For each feature point information group in the preset candidate queue, perform the following aggregation processing: For a first feature point information group in the preset candidate queue, each piece of three-dimensional point cloud information corresponding to each piece of feature point information in the first feature point information group in the three-dimensional point cloud information set is determined as a first target three-dimensional point cloud information set; Determine the first feature point information group as the common view feature point information group; Determine each feature point information group except the first feature point information group in the preset candidate queue as a target feature point information group set; For each target feature point information group in the target feature point information group set, perform the following steps: Determine each piece of three-dimensional point cloud information in the three-dimensional point cloud information set corresponding to each piece of target feature point information in the target feature point information group as a second target three-dimensional point cloud information set; In response to determining that the amount of three-dimensional point cloud information in the intersection of the first target three-dimensional point cloud information set and the second target three-dimensional point cloud information set is greater than a preset threshold, determining the target feature point information group as a common view feature point information group; Determining each determined common view feature point information group as a common view feature point information group set; Deleting the first feature point information group from the preset candidate queue to update the preset candidate queue; In response to determining that the preset candidate queue is not empty, performing the aggregation process again according to the updated preset candidate queue; Based on the image feature point information set of the object to be located and the respective common view feature point information group sets, at least three image feature point information of the object to be located in the image feature point information set of the object to be located are matched as respective two-dimensional feature point information with the three-dimensional point cloud information in the three-dimensional point cloud information set to obtain respective matching feature point information groups, wherein the matching feature point information groups in the respective matching feature point information groups include one image feature point information of the object to be located and one three-dimensional point cloud information; Based on the respective matching feature point information groups, relative position information between the positioning object in the image of the object to be positioned and the camera device is generated.
2. The method according to claim 1, wherein: The extracting process of feature point information on the object view image to obtain an initial feature point information group corresponding to the positioned object includes: Performing background removal processing on the object perspective image to obtain a background-removed object perspective image; The background-removed object viewing angle map is input into a pre-trained feature point extraction model to obtain an initial feature point information group corresponding to the positioned object.
3. The method according to claim 1, wherein: The feature point matching prediction model includes: a feature vector generation layer, an attention mechanism layer, a multi-layer perceptron layer and an output layer; and The step of inputting the initial feature point information set into a pre-trained feature point matching prediction model to obtain each matching feature point prediction information set includes: Inputting the initial feature point information set into the feature vector generation layer to obtain a feature vector set, wherein one initial feature point information set in the initial feature point information set corresponds to one corresponding feature vector set in the feature vector set; Inputting the feature vector set into the attention mechanism layer, generating correlation weight information between every two feature vectors that are not in the same group in the feature vector set, and obtaining each correlation weight information; Inputting each correlation weight information into the multi-layer perceptron layer to obtain each matching feature point prediction information group information; Input the information of each matching feature point prediction information group into the output layer to obtain each matching feature point prediction information group.
4. The method according to claim 1, wherein: The method of matching at least three pieces of image feature point information of the object to be located in the image feature point information set of the object to be located as pieces of two-dimensional feature point information with three-dimensional point cloud information in the three-dimensional point cloud information set based on the image feature point information set of the object to be located and the pieces of common view feature point information groups to obtain pieces of matching feature point information groups includes: Based on the image feature point information set of the object to be located and the respective common view feature point information group sets, retrieving the common view feature point information group with the highest feature similarity with the image feature point information set of the object to be located from the respective common view feature point information group sets as the target common view feature point information group; Based on the image feature point information set of the object to be located, the target common view feature point information group and the three-dimensional point cloud information set, each matching feature point information group is generated.
5. The method according to claim 4, wherein: The generating each matching feature point information group based on the image feature point information set of the object to be located, the target common view feature point information group and the three-dimensional point cloud information set comprises: Inputting the image feature point information set of the object to be located and the target common view feature point information group into a pre-trained image feature point matching model to obtain each initial matching feature point information group, wherein each initial matching feature point information group in each initial matching feature point information group includes one image feature point information of the object to be located and one feature point information; For each of the initial matching feature point information groups, the following steps are performed: Determining the feature point information included in the initial matching feature point information group as the feature point information to be replaced; Determine the three-dimensional point cloud information corresponding to the feature point information to be replaced as replacement point cloud information; Updating the feature point information included in the initial matching feature point information group to the replacement point cloud information, so as to update the initial matching feature point information group; The updated initial matching feature point information group is determined as the matching feature point information group.
6. A device for generating relative posture information of a camera device, comprising: A first acquisition unit is configured to acquire an object perspective image set corresponding to the positioned object, wherein each object perspective image in the object perspective image set corresponds to each shooting perspective; A first extraction processing unit is configured to perform feature point information extraction processing on each object perspective image in the object perspective image set to obtain an initial feature point information group corresponding to the positioning object; A first determining unit is configured to determine each obtained initial feature point information group as an initial feature point information group set, wherein an object perspective image in the object perspective image set corresponds to a corresponding initial feature point information group in the initial feature point information group set; An input unit, configured to input the initial feature point information group set into a pre-trained feature point matching prediction model to obtain each matching feature point prediction information group; A first generating unit is configured to generate a three-dimensional point cloud information set corresponding to each matching feature point prediction information group based on each matching feature point prediction information group; The elimination processing unit is configured to, for each initial feature point information group in the initial feature point information group set, perform an invalid feature point information elimination process on the initial feature point information group according to the respective matching feature point prediction information groups to obtain a feature point information group, wherein the performing an invalid feature point information elimination process on the initial feature point information group according to the respective matching feature point prediction information groups to obtain a feature point information group comprises: determining each matching feature point prediction information included in each matching feature point prediction information group in the respective matching feature point prediction information groups as each target matching feature point prediction information; determining each determined target matching feature point prediction information as a target matching feature point prediction information set; in response to determining that there is initial feature point information in the initial feature point information group that is different from the target matching feature point prediction information in the target matching feature point prediction information set, deleting each initial feature point information that is different from the target matching feature point prediction information from the initial feature point information group to update the initial feature point information group; and determining the updated initial feature point information group as the feature point information group; A second determining unit is configured to determine the obtained feature point information groups as a feature point information group set, wherein one feature point information group in the feature point information group set corresponds to a corresponding piece of three-dimensional point cloud information in the three-dimensional point cloud information set, and each feature point information in the feature point information group corresponds to the three-dimensional point cloud information; A second acquisition unit is configured to acquire an image of the object to be positioned corresponding to the positioning object captured by the camera device; A second extraction processing unit is configured to perform feature point information extraction processing on the image of the object to be located, and generate a feature point information set of the image of the object to be located corresponding to the image of the object to be located; The second generating unit is configured to generate each common view feature point information set based on the image feature point information set of the object to be located, the feature point information set and the three-dimensional point cloud information set, wherein the generating each common view feature point information set based on the image feature point information set of the object to be located, the feature point information set and the three-dimensional point cloud information set includes: determining the number of feature point information included in each feature point information group in the feature point information set as a sorting quantity to obtain each sorting quantity, wherein one of the sorting quantities corresponds to a feature point information group corresponding to the feature point information set in the feature point information set; according to the sorting quantities, storing each feature point information group in the feature point information set into a preset candidate queue; performing the following aggregation processing on each feature point information group in the preset candidate queue: for the first feature point information group in the preset candidate queue, storing each three-dimensional point information set corresponding to each feature point information in the first feature point information group in the three-dimensional point cloud information set The method comprises the following steps: determining cloud information as a first target three-dimensional point cloud information set; determining the first feature point information group as a common view feature point information group; determining each feature point information group except the first feature point information group in the preset candidate queue as a target feature point information group set; for each target feature point information group in the target feature point information group set, performing the following steps: determining each three-dimensional point cloud information corresponding to each target feature point information in the target feature point information group in the three-dimensional point cloud information set as a second target three-dimensional point cloud information set; in response to determining that the number of three-dimensional point cloud information in the intersection of the first target three-dimensional point cloud information set and the second target three-dimensional point cloud information set is greater than a preset threshold, determining the target feature point information group as a common view feature point information group; determining each determined common view feature point information group as a common view feature point information group set; deleting the first feature point information group from the preset candidate queue to update the preset candidate queue; in response to determining that the preset candidate queue is not empty, performing the aggregation processing again according to the updated preset candidate queue; A matching unit is configured to match at least three pieces of image feature point information of the object to be located in the image feature point information set of the object to be located as pieces of two-dimensional feature point information with three-dimensional point cloud information in the three-dimensional point cloud information set based on the image feature point information set of the object to be located and the pieces of common view feature point information group sets, to obtain pieces of matching feature point information groups, wherein the matching feature point information groups in the pieces of matching feature point information groups include one piece of image feature point information of the object to be located and one piece of three-dimensional point cloud information; The third generating unit is configured to generate relative position information of the positioning object and the camera device in the image of the object to be positioned based on the respective matching feature point information groups.
7. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.
8. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.
