A method and device for determining the position and posture of a camera

Abstract

The present invention discloses a method and device for determining the position and posture of a camera, which relates to the field of computer technology, in order to solve the problem of how to use easily accessible information to enhance scene understanding capabilities and improve the robustness of visual positioning. The method for determining the position and posture of a camera, the camera is used to collect images, and the images include multiple frames of images; the method includes: tracking the target position of the image to obtain the inter-frame relationship of each target, and the inter-frame relationship of each target is the relationship between each target in the current frame image and the next frame image; fusing the inter-frame relationship of each target into the global information of the current frame image to obtain the fusion information of the current frame image; and using the fusion information to determine the position and posture information of the camera. The method for determining the position and posture of a camera provided by the present invention can enhance the scene understanding capabilities and improve the robustness and accuracy of visual positioning.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for determining the position and posture of a camera. Background Art

[0002] In related technologies, as visual positioning technology has entered the stage of robust perception, people have been committed to researching methods to perform high-level scene perception and understanding to increase the robustness of visual positioning tasks. Currently, most methods mainly use semantic understanding to link semantic concepts, such as object classification, material composition, etc., with the geometric structure of the environment, but this method often requires a large amount of annotation information such as semantic segmentation, which is too costly in large-scale scenarios. Summary of the invention

[0003] The purpose of the present invention is to provide a method for determining the position and posture of a camera, which is used to solve the problem of how to use easily accessible information to enhance scene understanding capabilities and improve the robustness of visual positioning.

[0004] In a first aspect, the present invention provides a method for determining the position and posture of a camera, wherein the camera is used to collect images, and the images include multiple frames of images; the method comprises:

[0005] Tracking the target position of the image to obtain the inter-frame relationship of each target, wherein the inter-frame relationship of each target is the relationship between each target in the current frame image and the next frame image;

[0006] The inter-frame relationship of each target is integrated into the global information of the current frame image to obtain the integrated information of the current frame image;

[0007] The fusion information is used to determine the position and posture information of the camera.

[0008] Compared with the prior art, in the method for determining the position and posture of a camera provided by the present invention, the camera is used to track the two frames of images before and after each target to obtain the inter-frame relationship of each target. This relationship reflects the motion state of each target in the scene. The inter-frame relationship of each target is combined with the global information extracted from the current frame image, and the obtained fusion information can not only reflect the position information of each target in the current frame image, but also reflect the position-related information of each target in the next frame image, thereby ensuring that the fusion information contains rich three-dimensional real scenes and enhancing the ability to understand three-dimensional real scenes. At the same time, this ability to understand three-dimensional real scenes can provide more robust fusion information for the camera's position and posture estimation, thereby improving the robustness and accuracy of visual positioning.

[0009] In a second aspect, the present invention also provides a camera posture determination device, comprising a processor and a communication interface coupled to the processor; the processor is used to run a computer program or instruction to implement any of the camera posture determination methods described above.

[0010] Compared with the prior art, the beneficial effects of the camera posture determination device provided by the present invention are the same as the beneficial effects of the camera posture determination method described in the above technical solution, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0012] Figure 1A A flowchart of a method for determining a camera's posture provided by an embodiment of the present invention is shown in FIG1 ;

[0013] Figure 1B A schematic diagram of the effect of position changes of various targets in the current frame image and the next frame image provided by an embodiment of the present invention;

[0014] Figure 2A A schematic diagram of the effect of using a target frame to mark a target according to an embodiment of the present invention;

[0015] Figure 2B A schematic diagram of the effect of the distance between objects in the current frame image provided by an embodiment of the present invention;

[0016] Figure 3 A schematic diagram of the effect of the distance between each target in the next frame of image provided by an embodiment of the present invention;

[0017] Figure 4 A schematic diagram of the effect of the changes of various objects in the current frame image between two corresponding frame images provided by an embodiment of the present invention;

[0018] Figure 5 A schematic diagram of the effect of the fusion processing process provided by an embodiment of the present invention;

[0019] Figure 6 A second flow chart of a method for determining a camera's posture provided by an embodiment of the present invention;

[0020] Figure 7 A structural block diagram of a device for determining the position and posture of a camera provided in an embodiment of the present invention;

[0021] Figure 8 A schematic diagram of the hardware structure of a device for determining the position and posture of a camera provided in an embodiment of the present invention;

[0022] Fig. 9 A schematic diagram of the structure of a chip provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. For example, the first threshold and the second threshold are only used to distinguish different thresholds, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0024] It should be noted that, in the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0025] In the present invention, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b and c, where a, b, c can be single or multiple.

[0026] In related technologies, as visual positioning technology has entered the stage of robust perception, people have been committed to researching methods to perform high-level scene perception and understanding to increase the robustness of visual positioning tasks. Currently, most methods mainly use semantic understanding to link semantic concepts, such as object classification, material composition, etc., with the geometric structure of the environment, but this method often requires a large amount of annotation information such as semantic segmentation, which is too costly in large-scale scenarios.

[0027] In response to the above technical problems, an embodiment of the present invention proposes a method for determining the position and posture of a camera, which can solve the problem of how to use easily accessible information to enhance scene understanding capabilities and improve the robustness of visual positioning, thereby enhancing scene understanding capabilities and improving the robustness and accuracy of visual positioning.

[0028] The camera posture determination method provided in the embodiment of the present invention is applied to a camera posture determination device, wherein the camera is used to collect images, and the images include multiple frames of images; Figure 1A As shown, the above method includes:

[0029] Step 101: Track the target position of the image to obtain the inter-frame relationship of each target.

[0030] Here, a camera is used to capture images of a three-dimensional real scene to obtain multiple frames of images. Target detection is performed on each frame of the multiple frames to obtain the targets contained in each frame, and the target positions of the targets contained in the current frame are tracked to obtain the relationship between the current frame and the next frame, that is, the inter-frame relationship of the targets. The inter-frame relationship of the targets reflects the motion state of the targets in the three-dimensional real scene; each frame of the multiple frames contains multiple targets, which may be people or vehicles, etc.

[0031] like Figure 1B As shown, the target position is tracked for the current frame image 11 to obtain the next frame image 12. The positions of the targets included in the current frame image 11 and the next frame image 12 change.

[0032] For example: perform target detection on the current frame image, and obtain the targets contained in the current frame image as: target 1, target 2, target 3; perform target position tracking on target 1, target 2, target 3, and obtain the relationship between target 1 in the current frame image and the next frame image, the relationship between target 2 in the current frame image and the next frame image, and the relationship between target 3 in the current frame image and the next frame image, that is, the inter-frame relationship of target 1, the inter-frame relationship of target 2, and the inter-frame relationship of target 3.

[0033] In practical applications, when performing target detection, a target detection framework can be used for detection. The target detection framework used in the present invention is faster-rcnn. It should be noted that any target detection framework with good performance can detect the target, and the present invention does not impose any limitation on the target detection framework.

[0034] In practical applications, when tracking the position of each target, a target tracking framework can be used. The above target tracking framework can use a discriminative scale space tracker (DSST) algorithm. DSST is an improvement based on the kernel correlation filter (KCF) algorithm. On the basis of KCF position tracking, the operation of scale tracking is added to ensure that when the target scale changes with the change of distance, the center point of the target frame can still reflect the position of the object in the scene. It should be noted that the present invention does not impose any restrictions on the target tracking framework used, as long as the position of the target can be tracked.

[0035] Step 102: Fusing the inter-frame relationship of the targets into the global information of the current frame image to obtain fusion information of the current frame image.

[0036] Information is extracted from the current frame image to obtain global information of the current frame image. It should be noted that extracting the global information of the current frame image is a prior art and will not be described in detail in the embodiment of the present invention.

[0037] When the inter-frame relationship of each target is obtained, the inter-frame relationship of each target is fused with the global information of the current frame image to obtain the fused information of the current frame image. For example, in the same example as above, the inter-frame relationship of target 1, target 2, and target 3 is fused with the global information of the current frame image to obtain the fused information.

[0038] Step 103: Determine the position information of the camera using the fusion information.

[0039] After obtaining the fusion information, the pose estimation network is used to process the fusion information to determine the pose information of the camera, thereby realizing the visual positioning of the camera. Among them, the pose estimation network can include: supervised learning model and unsupervised learning model; supervised learning models include: posenet, deepvo, etc.; unsupervised learning models include: sfmlearner, etc.

[0040] In the embodiment of the present invention, the camera is used to track the two frames of images before and after each target to obtain the inter-frame relationship of each target; the inter-frame relationship of each target is the inter-frame position relationship feature established between each target extracted from two adjacent two-dimensional images, and this relationship feature reflects the motion state of each target in the scene. The inter-frame relationship of each target is combined with the global information extracted from the current frame image to obtain fusion information, which enhances the ability to understand the three-dimensional real scene. At the same time, this ability to understand the three-dimensional real scene can provide more robust fusion information for the camera's pose estimation, thereby improving the robustness and accuracy of visual positioning.

[0041] In one embodiment, the target position tracking of the image to obtain the inter-frame relationship of each target includes: using a target frame to determine the distance between each target in each frame of the image; with the assistance of the target frame, based on the distance between each target in the frame of the current frame and the distance between each target in the frame of the next frame, determining the inter-frame relationship between each target.

[0042] When determining the inter-frame relationship of each target, the target contained in each frame is identified by using the target frame, and the distance between each target frame in each frame is used as the intra-frame target distance of each frame. The intra-frame target distance can be represented in matrix form.

[0043] like Figure 2A As shown, the target included in the current frame image 201 is marked using a target frame.

[0044] like Figure 2B As shown, target detection is performed on the current frame image to obtain target 1, target 2, target 3, target 4, and target 5; target 1, target 2, target 3, target 4, and target 5 are marked using target frame 20, target frame 21, target frame 22, target frame 23, and target frame 24, respectively; the distance d1 between target frame 20 and target frame 21, the distance d2 between target frame 21 and target frame 22, the distance d3 between target frame 22 and target frame 23, the distance d4 between target frame 23 and target frame 24, the distance d5 between target frame 24 and target frame 20, etc. are used as the distances between targets within the frame of the current frame image.

[0045] like Figure 3 As shown, target detection is performed on the next frame image to obtain target 1, target 2, target 3, target 4, and target 5; target 1, target 2, target 3, target 4, and target 5 are marked using target frame 25, target frame 26, target frame 27, target frame 28, and target frame 29, respectively; the distance d6 between target frame 25 and target frame 26, the distance d7 between target frame 26 and target frame 27, the distance d8 between target frame 27 and target frame 28, the distance d9 between target frame 28 and target frame 29, the distance d10 between target frame 29 and target frame 25, etc. are used as the distances between targets within the frame of the next frame image.

[0046] With the assistance of each target frame in the current frame image and each target frame in the next frame image, the inter-frame relationship of each target is determined according to the distance between each target in the frame of the current frame image and the distance between each target in the frame of the next frame image.

[0047] In one embodiment, the distance between each target in each frame of the image is the distance between geometric centers of each target frame of the corresponding frame image.

[0048] Here, the distance between each target in each frame of the image is the distance between the geometric centers of each target frame of the corresponding frame image. As shown in FIG2 , the distance d1 between the geometric center of the target frame 20 of the current frame image and the geometric center of the target frame 21, the distance d2 between the geometric center of the target frame 21 and the geometric center of the target frame 22, the distance d3 between the geometric center of the target frame 22 and the geometric center of the target frame 23, the distance d4 between the geometric center of the target frame 23 and the geometric center of the target frame 24, the distance d5 between the geometric center of the target frame 24 and the geometric center of the target frame 20, etc., constitute the distance between each target in the frame of the current frame image. The distance between each target in the frame of the next frame image is similar to the distance between each target in the frame of the current frame image, and the present invention will not repeat them.

[0049] The distances between the objects in the frame are expressed in matrix form. The expression of the distances between the objects in the frame of the current frame image box(t) is shown in formula (1):

[0050]

[0051] Among them, d i,j Represents the distance between the geometric center of the i-th target box and the geometric center of the j-th target box.

[0052] In one embodiment, with the assistance of the target frame, the inter-frame relationship of each target is determined according to the distance between each target in the frame of the current frame image and the distance between each target in the frame of the next frame image, including: determining the position information of each target frame in the frame of each frame of the image; determining the inter-frame relationship of each target according to the position information of each target frame in the frame of each frame of the image and the distance between each target in the frames of two adjacent frames of the image.

[0053] When determining the inter-frame relationship of each target, first determine the position information of each target frame in each frame of the image. The position information of each target frame in the frame can be expressed in a matrix form; as shown in FIG2 , the position information of the target frame 20 included in the current frame image is determined as position information 1, the position information of the target frame 21 is determined as position information 2, the position information of the target frame 22 is determined as position information 3, the position information of the target frame 23 is determined as position information 4, and the position information of the target frame 24 is determined as position information 5. Figure 3 As shown, the position information of the target frame 25 included in the next frame image is determined to be position information 6, the position information of the target frame 26 is determined to be position information 7, the position information of the target frame 27 is determined to be position information 8, the position information of the target frame 28 is determined to be position information 9, and the position information of the target frame 29 is determined to be position information 10.

[0054] The inter-frame relationship of each target is determined according to the position information of each target frame in the current frame image, the distance between each target in the current frame image and the distance between each target in the next frame image.

[0055] In one embodiment, the position information of each target frame in each frame of the image includes: coordinates of two diagonal points of each target frame in the corresponding frame image that is in a diagonal state.

[0056] Here, the position information of each target frame in each frame of the image includes: the coordinates of two diagonal points of each target frame in the corresponding frame of the image, such as: the coordinates of the upper left corner point (x1, y1) and the coordinates of the lower right corner point (x2, y2). When each frame of the image contains multiple target frames, the coordinates of the two diagonal points of the multiple target frames can be represented in matrix form.

[0057] In one embodiment, the inter-frame relationship of each target is determined based on the position information of each target frame in each frame of the image and the distance between each target in the frames of two adjacent frames of the image, including: determining the change information of each target between the corresponding two frames of the image based on the position information of each target frame in the two adjacent frames of the image; determining the inter-frame relationship of each target based on the position information of each target frame in each frame of the image, the distance between each target in the frames of two adjacent frames of the image, and the change information of each target between the corresponding two frames of the image.

[0058] Here, the inter-frame relationship of each target reflects the state of each target in the current frame image. After obtaining the distance between each target in the current frame image, the target tracking framework is used to track the new position of the target contained in the next frame image, and the distance between each target in the next frame image is established. At the same time, it is also necessary to determine the change information of each target between two adjacent frames, that is, the change between the position of the target frame corresponding to each target in the current frame image and the position of the target frame corresponding to each target in the next frame image between the previous and next frames.

[0059] When determining the change information of each target between two adjacent frames, it is determined based on the position information of each target frame of the two adjacent frames. The distance between the geometric center of each target frame included in the current frame image and the geometric center of the corresponding target frame included in the next frame image is used as the change information of each target between the corresponding two frames.

[0060] like Figure 4 As shown, the current frame image includes a target frame 24, and the next frame image includes a target frame 40. The target frame 24 and the target frame 40 identify the same target: target 1. The distance d1 between the geometric center of the target frame 24 and the geometric center of the target frame 40 is used as the change information of the target 1 between the corresponding two frame images.

[0061] After obtaining the change information of each target between the corresponding two frames of images, the inter-frame relationship of each target is determined according to the position information of each target frame in the frame of the current frame image, the distance between each target in the frame of the current frame image, the distance between each target in the frame of the next frame image and the change information of each target between the corresponding two frames of images.

[0062] In one embodiment, the inter-frame relationship of each target includes:

[0063] G(t)=box t +box(t)+box(t+1)+box(t,t+1)Formula (2);

[0064] Among them, box t is the position information of each target box in the current frame image, box(t) and box(t+1) are the distances between each target in the current frame image and the next frame image respectively, and box(t,t+1) is the change information of each target between the corresponding two frames of images.

[0065] Here, box t The expression of is shown in formula (3):

[0066]

[0067] Among them, (x i,1 ,y i,1 ), (x i,2 ,y i,2 ) are the coordinates of the upper left corner and lower right corner of the i-th target box respectively.

[0068] Here, the expression of box(t,t+1) is shown in formula (4):

[0069]

[0070] Among them, d i is the position change of the i-th target in the t-th frame image and the t+1-th frame image, expressed as the difference.

[0071] In one embodiment, the inter-frame relationship of the targets is fused into the global information of the current frame image to obtain the fused information of the current frame image, including: normalizing the inter-frame relationship of the targets to obtain the normalized inter-frame relationship; performing dimensionality reduction processing on the normalized inter-frame relationship and the global information of the current frame image; and splicing the normalized inter-frame relationship after dimensionality reduction and the global information of the current frame image after dimensionality reduction to obtain fused information.

[0072] Here, after obtaining the inter-frame relationship of each target, in order to make the inter-frame relationship of each target and the global information of the current frame image at the same order of magnitude, the inter-frame relationship of the above targets is normalized to obtain the normalized inter-frame relationship.

[0073] In practical applications, when normalizing the inter-frame relationship of each target, the tf.nn.l2_normalize(*) function may be used for normalization. It should be noted that the normalization method used in the present invention can be used as long as it can achieve normalization, and the embodiment of the present invention does not impose any limitation on the normalization method.

[0074] After obtaining the normalized inter-frame relationship, the normalized inter-frame relationship and the global information of the current frame image are respectively subjected to dimensionality reduction processing to obtain the normalized inter-frame relationship after dimensionality reduction and the global information of the current frame image after dimensionality reduction. In practical applications, when performing dimensionality reduction processing, the flatten(*) function can be used to perform dimensionality reduction processing on the normalized inter-frame relationship and the global information of the current frame image to obtain a one-dimensional normalized inter-frame relationship and a one-dimensional global information of the current frame image.

[0075] After obtaining the normalized inter-frame relationship after dimensionality reduction and the global information of the current frame image after dimensionality reduction, the normalized inter-frame relationship after dimensionality reduction and the global information of the current frame image after dimensionality reduction are spliced ​​to obtain fusion information. When performing the splicing process, the concat(*) function can be used.

[0076] like Figure 5 As shown, the t-th frame image 30 and the t+1-th frame image 31 are processed by a relational feature extractor 32 to obtain the inter-frame relationship of each target, and the inter-frame relationship of each target is normalized to obtain the normalized inter-frame relationship; and processed by a dimensionality reduction function 33 to obtain the normalized inter-frame relationship after dimensionality reduction; the t-th frame image 30 is processed by a global feature extractor 34 to obtain the global information of the current frame image; and processed by a dimensionality reduction function 35 to obtain the global information of the current frame image after dimensionality reduction. The normalized inter-frame relationship after dimensionality reduction and the global information of the current frame image after dimensionality reduction are spliced ​​by a splicing function 36 to obtain fusion information.

[0077] In one embodiment, the pose information of the camera includes: a translation matrix and a rotation matrix; the use of the fusion information to determine the pose information of the camera includes: using a pose estimation network to process the fusion information to determine the dynamic features and static features in the fusion information; setting the weight of the dynamic feature to be smaller than the weight of the static feature; and determining the translation matrix and rotation matrix of the camera based on the dynamic feature, the weight of the dynamic feature, the static feature, and the weight of the static feature.

[0078] Here, after obtaining the fusion information, the fusion information is processed using the pose estimation network to determine the dynamic features and static features in the fusion information. At the same time, the weights of the dynamic features and the static features are adjusted to reduce the weights of the dynamic features and increase the weights of the static features, so that the weights of the dynamic features are less than the weights of the static features; further, the pose information of the camera is determined based on the dynamic features, the weights of the dynamic features, the static features, and the weights of the static features; the pose information includes: a translation matrix and a rotation matrix. It should be noted that determining the pose information of the camera based on the dynamic features, the weights of the dynamic features, the static features, and the weights of the static features belongs to the prior art and will not be repeated in the present invention.

[0079] The present invention sets the weight of dynamic features to be smaller than that of static features, so that the weight of dynamic features will gradually decrease during training, that is, the impact of dynamic features on the system is reduced, further improving the robustness of the system. This also avoids noise such as edge sharpening caused by directly removing dynamic targets.

[0080] The present invention also provides a method for determining the position and posture of a camera, such as Figure 6 As shown, the above method includes:

[0081] Step 601: obtaining a current frame image and identifying a target contained in the current frame image;

[0082] Step 602: Determine the distance between each target in the current frame image;

[0083] Step 603: tracking each target in the current frame image to obtain the distance between each target in the next frame image;

[0084] Step 604: With the aid of the target frame, determine the inter-frame relationship of each target according to the distance between each target in the current frame image and the distance between each target in the next frame image;

[0085] Step 605: Fusing the inter-frame relationship of each target with the global information of the current frame image to obtain fusion information;

[0086] Step 606: Obtain the camera's position and posture information based on the fusion information.

[0087] The embodiments of the present invention can achieve the following technical effects:

[0088] The present invention marks the extracted targets such as cars and people on a two-dimensional image, establishes the intra-frame distance between the targets, tracks the targets of the two frames before and after, and obtains the inter-frame relationship of the targets of the two frames before and after. This feature reflects the position change relationship between the targets in the current scene, enhances the understanding of the overall state of the scene in the three-dimensional scene, and at the same time, the change of the target position relationship during the scene change also reflects the motion state of each target. After normalizing and reducing the inter-frame relationship of each target, it is fused with the global information to obtain fused information, wherein the global information realizes the global understanding of the scene. The global information is processed to obtain dynamic features and static features, which can distinguish the different states of dynamic targets and static targets to a certain extent; the present invention adjusts the weight of dynamic features and the weight of static features under the drive of deep network data, reduces the weight of dynamic features or even removes them. Under the drive of a large amount of data, the weight of dynamic features will gradually decrease during training, that is, the influence of dynamic features on the system is reduced, and the robustness of the system is further improved, which also avoids the noise such as edge sharpening caused by directly removing dynamic targets. Therefore, while improving the scene understanding ability, it is also possible to remove the features of targets with large changes in motion state to a certain extent, thereby improving the robustness of the system in dynamic scenes and ultimately achieving improved visual positioning accuracy. That is, the present invention can enhance the robustness and counteract the ambiguity of dynamic and static targets to a certain extent.

[0089] In the case of dividing each functional module into corresponding functional modules, Figure 7 FIG. 2 shows a structural block diagram of a device for determining the position and posture of a camera provided by an embodiment of the present invention. Figure 7 As shown, the camera posture determination device 70 includes: a communication module 701 and a processing module 702.

[0090] The communication module 701 is used to obtain images captured by the camera.

[0091] The processing module 702 is used to support the camera posture determination device to execute steps 101 to 103 in the above embodiment.

[0092] All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0093] In some possible implementations, the above-mentioned camera posture determination device may further include a storage module 703 for storing program codes and data of the base station.

[0094] Among them, the processing module can be a processor or a controller, for example, a central processing unit (CPU), a general processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of the present invention. The processor can also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The communication module can be a transceiver, a transceiver circuit or a communication interface, and the like. The storage module can be a memory.

[0095] When the processing module is a processor, the communication module is a communication interface, and the storage module is a memory, the camera posture determination device involved in the embodiment of the present invention can be Figure 8 The camera pose determination device is shown.

[0096] Figure 8 FIG. 1 is a schematic diagram showing the hardware structure of a device for determining the position and posture of a camera provided by an embodiment of the present invention. Figure 8 As shown, the camera posture determination device 80 includes a processor 801 and a communication interface 802 .

[0097] like Figure 8 As shown, the processor may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present invention. The communication interface may be one or more. The communication interface may use any transceiver-like device for communicating with other devices or communication networks.

[0098] like Figure 8 As shown, the above-mentioned camera posture determination device may further include a communication line 803. The communication line may include a channel to transmit information between the above-mentioned components.

[0099] Optional, such as Figure 8As shown, the camera posture determination device may further include a memory 804. The memory is used to store computer execution instructions for executing the solution of the present invention, and the execution is controlled by the processor. The processor is used to execute the computer execution instructions stored in the memory, thereby implementing the method provided by the embodiment of the present invention.

[0100] like Figure 8 As shown, the above-mentioned memory can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory can exist independently and be connected to the processor through a communication line. The memory can also be integrated with the processor.

[0101] Optionally, the computer-executable instructions in the embodiment of the present invention may also be referred to as application program codes, which is not specifically limited in the embodiment of the present invention.

[0102] In a specific implementation, as an example, Figure 8 As shown, the processor 801 may include one or more CPUs, such as Figure 8 CPU0 and CPU1 in.

[0103] In a specific implementation, as an example, Figure 8 As shown, the camera posture determination device may include multiple processors, such as Figure 8 Each of these processors may be a single-core processor or a multi-core processor.

[0104] Fig. 9 Schematic diagram of the structure of the chip provided by the embodiment of the present invention. Fig. 9 As shown, the chip 90 includes one or more (including two) processors 801 and a communication interface 802 .

[0105] Optional, such as Fig. 9As shown, the chip also includes a memory 804, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory (NVRAM).

[0106] In some embodiments, Fig. 9 As shown, the memory stores the following elements, execution modules or data structures, or their subsets, or their extended sets.

[0107] In the embodiment of the present invention, Fig. 9 As shown, the corresponding operation is performed by calling the operation instruction stored in the memory (the operation instruction may be stored in the operating system).

[0108] like Fig. 9 As shown, the processor controls the processing operations of any one of the camera's posture determination devices, and the processor may also be referred to as a central processing unit (CPU).

[0109] like Fig. 9 As shown, the memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include NVRAM. For example, in an application, the memory, the communication interface, and the memory are coupled together through a bus system, wherein the bus system may include a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, in Fig. 9 Various buses are labeled as bus system 805 .

[0110] like Fig. 9As shown, the method disclosed in the above embodiment of the present invention can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an ASIC, a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0111] In one possible implementation, Fig. 9 As shown, the communication interface is used to obtain the image captured by the camera. The processor is used to execute steps 101 to 103 of the method for determining the position and posture of the camera in the embodiment shown in FIG1 .

[0112] On the one hand, a computer-readable storage medium is provided, in which instructions are stored. When the instructions are executed, the functions performed by the camera posture determination device in the above embodiment are implemented.

[0113] On the one hand, a chip is provided, which is applied to a camera posture determination device. The chip includes at least one processor and a communication interface. The communication interface is coupled to at least one processor, and the processor is used to run instructions to implement the functions performed by the camera posture determination device in the above embodiments.

[0114] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present invention is executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user device or other programmable device. The computer program or instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instruction can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a tape; it can also be an optical medium, such as a digital video disc (DVD); it can also be a semiconductor medium, such as a solid state drive (SSD).

[0115] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0116] Although the present invention has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present invention. Accordingly, this specification and the accompanying drawings are merely exemplary illustrations of the present invention as defined by the appended claims and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present invention. Obviously, those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, the present invention is intended to include such modifications and variations if they fall within the scope of the claims of the present invention and their equivalents.

Claims

1. A method for determining the position and posture of a camera, characterized in that: The camera is used to collect images, and the images include multiple frames of images; the method includes: Track the target position of the image to obtain the inter-frame relationship of each target, wherein the inter-frame relationship of each target is the relationship between each target in the current frame image and the next frame image; the inter-frame relationship of each target includes: G(t)=box t +box(t)+box(t+1)+box(t,t+1) Among them, box t is the position information of each target box in the current frame image, box(t) and box(t+1) are the distances between each target in the current frame image and the next frame image respectively, and box(t,t+1) is the change information of each target between the corresponding two frames of images; The inter-frame relationship of each target is integrated into the global information of the current frame image to obtain the integrated information of the current frame image; The fusion information is used to determine the position and posture information of the camera.

2. The method for determining the position and posture of a camera according to claim 1, characterized in that: Tracking the target position of the image to obtain the inter-frame relationship of each target includes: The target frame is used to determine the distance between each target in each frame of the image; With the aid of the target frame, the inter-frame relationship of the targets is determined according to the distance between the targets in the current frame image and the distance between the targets in the next frame image.

3. The method for determining the position and posture of a camera according to claim 2, characterized in that: The distance between each target in each frame of the image is the distance between geometric centers of each target frame of the corresponding frame image.

4. The method for determining the position and posture of a camera according to claim 2, characterized in that: The inter-frame relationship of each target is determined according to the distance between each target in the frame of the current frame image and the distance between each target in the frame of the next frame image with the assistance of the target frame, including: Determine the position information of each target frame in each frame of the image; The inter-frame relationship of each target is determined according to the position information of each target frame in each frame of the image and the distance between each target in two adjacent frames of the image.

5. The method for determining the position and posture of a camera according to claim 4, characterized in that: The position information of each target frame in each frame of the image includes: coordinates of two diagonal points of each target frame in the corresponding frame image that is in a diagonal state.

6. The method for determining the position and posture of a camera according to claim 4, characterized in that: Determining the inter-frame relationship of each target according to the position information of each target frame in each frame of the image and the distance between each target in two adjacent frames of the image includes: Determine the change information of each target between the two corresponding frames of images according to the position information of each target frame of two adjacent frames of images; The inter-frame relationship of each target is determined according to the position information of each target frame in each frame of the image, the distance between each target in two adjacent frames of the image, and the change information of each target between the corresponding two frames of the image.

7. The method for determining the position and posture of a camera according to claim 1, characterized in that: The step of fusing the inter-frame relationship of the targets into the global information of the current frame image to obtain the fusion information of the current frame image includes: Normalizing the inter-frame relationship of each target to obtain a normalized inter-frame relationship; Performing dimensionality reduction processing on the normalized inter-frame relationship and the global information of the current frame image; The normalized inter-frame relationship after dimensionality reduction and the global information of the current frame image after dimensionality reduction are spliced ​​to obtain fusion information.

8. The method for determining the position and posture of a camera according to claim 1, characterized in that: The camera's posture information includes: a translation matrix and a rotation matrix; and using the fusion information to determine the camera's posture information includes: Processing the fused information using a pose estimation network to determine dynamic features and static features in the fused information; Setting the weight of the dynamic feature to be smaller than the weight of the static feature; A translation matrix and a rotation matrix of the camera are determined according to the dynamic features, the weights of the dynamic features, the static features, and the weights of the static features.

9. A camera posture determination device, characterized in that: It comprises a processor and a communication interface coupled to the processor; the processor is used to run a computer program or instruction to implement the method for determining the position and posture of a camera as described in any one of claims 1 to 8.

Images