Method, device, electronic device and storage medium for detecting three-dimensional position of object

Abstract

The present disclosure relates to a method, device, electronic device and storage medium for detecting the three-dimensional position of an object. The method includes: determining the three-dimensional information of the object corresponding to each video frame in a target video, wherein the target video contains at least one object; clustering the three-dimensional information of the object corresponding to all video frames in the target video; screening all clusters corresponding to the target video according to the video frames associated with each cluster to obtain the target cluster corresponding to the target video; determining the initial three-dimensional position of the object corresponding to each target cluster according to the three-dimensional information of the object corresponding to each target cluster; determining the target three-dimensional position of the object corresponding to each target cluster according to the camera posture, the three-dimensional information of the object corresponding to each target cluster and the initial three-dimensional position of the object corresponding to each target cluster. According to the scheme of the present disclosure, more accurate detection of the three-dimensional position information of the object can be achieved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a method, device, electronic device and storage medium for detecting a three-dimensional position of an object. Background Art

[0002] With the increasing development of Mixed Reality (MR) technology, virtual scene information can be presented in real scenes to build an interactive feedback information loop between the real world, the virtual world and the user, so as to enhance the realism of the user experience.

[0003] At present, 3D reconstruction based on visual information is the basis of many mixed reality applications, such as 3D reconstruction of common objects in indoor environments. Furthermore, special effects can be superimposed on images or videos based on the 3D position information of the reconstructed objects to enrich the presentation of images or videos.

[0004] Therefore, there is an urgent need for a solution that can accurately obtain the three-dimensional position information of an object. Summary of the invention

[0005] The present disclosure provides a method, device, electronic device and storage medium for detecting the three-dimensional position of an object, so as to achieve the purpose of more accurate object position information detection by providing a three-dimensional position detection solution of an object with better robust performance and wider application range. The technical solution of the present disclosure is as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, a method for detecting a three-dimensional position of an object is provided, comprising:

[0007] Determine the three-dimensional information of the object corresponding to each video frame in the target video, wherein the target video contains at least one object; cluster the three-dimensional information of the object corresponding to all video frames in the target video; filter all clusters corresponding to the target video according to the video frames associated with each cluster to obtain the target cluster corresponding to the target video; determine the initial three-dimensional position of the object corresponding to each target cluster according to the three-dimensional information of the object corresponding to each target cluster; determine the target three-dimensional position of the object corresponding to each target cluster according to the camera posture, the three-dimensional information of the object corresponding to each target cluster and the initial three-dimensional position of the object corresponding to each target cluster.

[0008] Optionally, determining the three-dimensional information of the object corresponding to each video frame in the target video includes:

[0009] Determine the camera posture corresponding to each video frame in the target video; perform object detection on each video frame in the target video to obtain the object detection result corresponding to each video frame; and obtain the three-dimensional object information corresponding to each video frame based on the camera posture and object detection result corresponding to each video frame.

[0010] Optionally, the object detection result corresponding to each video frame includes the physical size of each object, the posture of each object relative to the camera, and the feature vector of each object; wherein, obtaining the three-dimensional information of the object corresponding to each video frame according to the camera posture and the object detection result corresponding to each video frame includes:

[0011] According to the physical size of each object, the posture of each object relative to the camera and the camera posture corresponding to each video frame, each object is projected into the world coordinate system to obtain the three-dimensional bounding box corresponding to each object; according to the feature vector of each object, a search is performed in the preset object model library to obtain the three-dimensional model corresponding to each object; the three-dimensional bounding box and the three-dimensional model corresponding to each object in each video frame are used as the three-dimensional information of the object corresponding to each video frame.

[0012] Optionally, determining the initial three-dimensional position of the object corresponding to each target cluster according to the three-dimensional information of the object corresponding to each target cluster includes:

[0013] According to the size of the 3D bounding box in each target cluster and the posture of the object relative to the camera, the initial 3D position of the object corresponding to each target cluster is determined.

[0014] Optionally, determining the initial three-dimensional position of the object corresponding to each target cluster according to the size of the corresponding three-dimensional bounding box in each target cluster and the posture of the object relative to the camera includes:

[0015] The average value of the sizes of all three-dimensional bounding boxes in each target cluster and the average value of the postures of all objects in each target cluster relative to the camera are projected into the world coordinate system to obtain the initial three-dimensional position of the object corresponding to each target cluster.

[0016] Optionally, the object detection result corresponding to each video frame also includes a first segmentation mask of each object; wherein, determining the target three-dimensional position of the object corresponding to each target cluster according to the camera posture, the three-dimensional information of the object corresponding to each target cluster, and the initial three-dimensional position of the object corresponding to each target cluster includes:

[0017] According to the camera posture and the initial three-dimensional position of the object corresponding to each target cluster, the three-dimensional model of the object corresponding to each target cluster is projected onto the video frame corresponding to each target cluster to obtain the second segmentation mask of the object corresponding to each target cluster; according to the first segmentation mask of each object, the second segmentation mask of the object corresponding to each target cluster, and the intersection and union ratio of the three-dimensional bounding box corresponding to each target cluster, the target three-dimensional position of the object corresponding to each target cluster is determined.

[0018] Optionally, determining the target three-dimensional position of the object corresponding to each target cluster according to the first segmentation mask of each object, the second segmentation mask of the object corresponding to each target cluster, and the intersection-over-union ratio of the three-dimensional bounding box corresponding to each target cluster includes:

[0019] According to the first segmentation mask of each object, the second segmentation mask of the object corresponding to each target cluster, and the intersection-and-union ratio of the three-dimensional bounding box corresponding to each target cluster, the objective function corresponding to all target clusters is determined, and the objective function is used to characterize the object position and posture; the objective function is minimized to obtain a result of the minimization processing, wherein the result of the minimization processing corresponds to the target three-dimensional position of the object corresponding to each target cluster.

[0020] Optionally, the objective function is: -log Loss ,in, Wherein, N represents the total number of the at least one object, M represents the total number of all video frames in the target video, Represents the first segmentation mask of the i-th object in the j-th frame video, represents the second segmentation mask of the i-th object in the j-th frame video, IOU represents intersection-over-union ratio, Represents the collision volume between objects; i is 1,…,N, j is 1,…,M.

[0021] Optionally, clustering the three-dimensional information of objects corresponding to all video frames in the target video includes:

[0022] The three-dimensional bounding boxes corresponding to all the video frames in the target video are clustered according to the intersection-and-union ratio between every two three-dimensional bounding boxes in the three-dimensional bounding boxes corresponding to all the video frames in the target video.

[0023] Optionally, screening all clusters corresponding to the target video according to the video frames associated with each cluster to obtain a target cluster corresponding to the target video includes:

[0024] Clusters whose associated video frames have a frame number less than a preset threshold among all clusters corresponding to the target video are eliminated to obtain a target cluster corresponding to the target video.

[0025] According to a second aspect of an embodiment of the present disclosure, there is provided a device for detecting a three-dimensional position of an object, comprising:

[0026] The first determination module is configured to determine the three-dimensional information of the object corresponding to each video frame in the target video, wherein the target video contains at least one object; the clustering module is configured to cluster the three-dimensional information of the object corresponding to all video frames in the target video; the screening module is configured to screen all clusters corresponding to the target video according to the video frames associated with each cluster, so as to obtain the target cluster corresponding to the target video; the second determination module is configured to determine the initial three-dimensional position of the object corresponding to each target cluster according to the three-dimensional information of the object corresponding to each target cluster; the third determination module is configured to determine the target three-dimensional position of the object corresponding to each target cluster according to the camera posture, the three-dimensional information of the object corresponding to each target cluster and the initial three-dimensional position of the object corresponding to each target cluster.

[0027] Optionally, the first determination module includes: a determination unit, a detection unit and a processing unit; wherein the determination unit is configured to determine the camera posture corresponding to each video frame in the target video; the detection unit is configured to perform object detection on each video frame in the target video to obtain the object detection result corresponding to each video frame; the processing unit is configured to obtain the three-dimensional object information corresponding to each video frame based on the camera posture and object detection result corresponding to each video frame.

[0028] Optionally, the processing unit, when the object detection result corresponding to each video frame includes the physical size of each object, the posture of each object relative to the camera and the feature vector of each object, is configured to specifically perform:

[0029] According to the physical size of each object, the posture of each object relative to the camera and the camera posture corresponding to each video frame, each object is projected into the world coordinate system to obtain the three-dimensional bounding box corresponding to each object; according to the feature vector of each object, a search is performed in the preset object model library to obtain the three-dimensional model corresponding to each object; the three-dimensional bounding box and the three-dimensional model corresponding to each object in each video frame are used as the three-dimensional information of the object corresponding to each video frame.

[0030] Optionally, the second determination module is configured to specifically perform: determining an initial three-dimensional position of an object corresponding to each target cluster according to a size of a three-dimensional bounding box within each target cluster and a posture of the object relative to a camera.

[0031] Optionally, the second determination module is configured to specifically perform: projecting the average value of the sizes of all three-dimensional bounding boxes within each target cluster and the average value of the postures of all objects within each target cluster relative to the camera into the world coordinate system to obtain the initial three-dimensional position of the object corresponding to each target cluster.

[0032] Optionally, the third determination module, when the object detection result corresponding to each video frame further includes the first segmentation mask of each object, is configured to specifically perform:

[0033] According to the camera posture and the initial three-dimensional position of the object corresponding to each target cluster, the three-dimensional model of the object corresponding to each target cluster is projected onto the video frame corresponding to each target cluster to obtain the second segmentation mask of the object corresponding to each target cluster; according to the first segmentation mask of each object, the second segmentation mask of the object corresponding to each target cluster, and the intersection and union ratio of the three-dimensional bounding box corresponding to each target cluster, the target three-dimensional position of the object corresponding to each target cluster is determined.

[0034] Optionally, the third determining module is configured to specifically execute:

[0035] According to the first segmentation mask of each object, the second segmentation mask of the object corresponding to each target cluster, and the intersection-and-union ratio of the three-dimensional bounding box corresponding to each target cluster, the objective function corresponding to all target clusters is determined, and the objective function is used to characterize the object position and posture; the objective function is minimized to obtain a result of the minimization processing, wherein the result of the minimization processing corresponds to the target three-dimensional position of the object corresponding to each target cluster.

[0036] Optionally, the objective function is: -log Loss ,in, Wherein, N represents the total number of the at least one object, M represents the total number of all video frames in the target video, Represents the first segmentation mask of the i-th object in the j-th frame video, represents the second segmentation mask of the i-th object in the j-th frame video, IOU represents intersection-over-union ratio, Represents the collision volume between objects; i is 1,…,N, j is 1,…,M.

[0037] Optionally, the clustering module is configured to specifically perform: clustering the 3D bounding boxes corresponding to all video frames in the target video according to the intersection-and-union ratio between every two 3D bounding boxes in the 3D bounding boxes corresponding to all video frames in the target video.

[0038] Optionally, the screening module is configured to specifically execute: eliminating clusters whose associated video frames have a frame number less than a preset threshold among all clusters corresponding to the target video, to obtain a target cluster corresponding to the target video.

[0039] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0040] A processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the three-dimensional position detection method of the object as described above.

[0041] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor in an electronic device, the electronic device can execute the three-dimensional position detection method of an object as described above.

[0042] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising at least one non-temporary computer-readable medium storing instructions translatable by at least one processor for implementing the aforementioned method for detecting a three-dimensional position of an object.

[0043] At least one of the above technical solutions adopted in the embodiments of the present disclosure can achieve the following beneficial effects:

[0044] When it is necessary to perform three-dimensional reconstruction on at least one object in a certain scene, the corresponding target video can be obtained, and the target video can be analyzed frame by frame to determine the three-dimensional information of the object corresponding to each video frame contained in the target video, so as to achieve the acquisition of the three-dimensional information of the object on the single-view image. Further, the three-dimensional information of the object corresponding to the multiple single-view images corresponding to the target video, that is, all the video frames contained in the target video, can be comprehensively analyzed, that is, by clustering the three-dimensional information of the object corresponding to all the video frames in the target video, and further filtering the clustering results, so that the initial three-dimensional position of the object corresponding to each target cluster can be first determined according to the three-dimensional information of the object corresponding to each filtered target cluster, and then the final target three-dimensional position of the object corresponding to each target cluster in the target video can be accurately determined according to the camera posture, the initial three-dimensional position corresponding to each target cluster and the three-dimensional information of the object. In this way, by providing a more robust performance, a wider range of applications and not limited to the need for rich texture information on the image, the purpose of more accurate object position information detection is achieved.

[0045] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:

[0047] Figure 1 The figure is a flow chart of a method for detecting a three-dimensional position of an object according to an exemplary embodiment.

[0048] Figure 2 The figure is a flow chart of a method for detecting a three-dimensional position of an object according to another exemplary embodiment.

[0049] Figure 3 is a schematic diagram of a three-dimensional bounding box of an object according to an exemplary embodiment.

[0050] Figure 4 is a schematic diagram of a three-dimensional bounding box of an object according to another exemplary embodiment.

[0051] Figure 5 The diagram is a schematic diagram before 3D model retrieval according to an exemplary embodiment.

[0052] Figure 6 Yes Figure 5 Schematic diagram of the results after 3D model retrieval.

[0053] Figure 7 The figure is a schematic diagram showing the object segmentation effect after object detection is performed on a video frame according to an exemplary embodiment.

[0054] Figure 8 The invention is a block diagram of a device for detecting a three-dimensional position of an object according to an exemplary embodiment.

[0055] Fig. 9 It is a block diagram of an electronic device according to an exemplary embodiment.

[0056] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure will be clearly and completely described below in combination with the specific embodiments of the present disclosure and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0058] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0059] The technical solutions provided by various embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings.

[0060] Embodiment 1

[0061] Please refer to Figure 1 , is a flow chart of a method for detecting a three-dimensional position of an object according to an exemplary embodiment, referring to Figure 1 The object three-dimensional position detection method may include the following steps:

[0062] S11, determining three-dimensional information of an object corresponding to each video frame in a target video, wherein the target video includes at least one object.

[0063] Optionally, the target video is obtained by shooting with a monocular camera, for example, shooting an indoor scene, which may include common objects such as tables, chairs, and sofas.

[0064] S13, clustering the three-dimensional information of objects corresponding to all video frames in the target video.

[0065] Further optionally, in this embodiment, the implementation process of the above S13 may include the following contents:

[0066] The three-dimensional bounding boxes corresponding to all the video frames in the target video are clustered according to the intersection-and-union ratio between every two three-dimensional bounding boxes in the three-dimensional bounding boxes corresponding to all the video frames in the target video.

[0067] Among them, the process of dividing a collection of physical or abstract objects into multiple classes consisting of similar objects is called clustering. The cluster generated by clustering is a collection of a group of data objects that are similar to objects in the same cluster and different from objects in other clusters. Cluster analysis, also known as group analysis, is a statistical analysis method for studying (sample or indicator) classification problems. Cluster analysis is very rich in content, including systematic clustering, ordered sample clustering, dynamic clustering, fuzzy clustering, graph clustering, cluster prediction, etc. When the intersection over union (IOU) between two three-dimensional bounding boxes is high, it means that the objects corresponding to the two three-dimensional bounding boxes are highly similar, and they can be classified into the same cluster. In other words, by using the IOU between the three-dimensional bounding boxes as a distance metric, accurate clustering of the three-dimensional bounding boxes corresponding to each video frame can be achieved, thereby achieving classification of each object contained in the target video.

[0068] S15, screening all clusters corresponding to the target video according to the video frames associated with each cluster, to obtain a target cluster corresponding to the target video.

[0069] Further optionally, in this embodiment, the implementation process of the above S15 may include the following contents:

[0070] The clusters whose associated video frames in all clusters corresponding to the target video are smaller than a preset threshold are eliminated to obtain the target cluster corresponding to the target video. In other words, the number of observations contained in each cluster (i.e., the total number of video frames) is obtained by counting, and clusters with too small number of observations are further filtered out based on the statistical results, so as to realize the screening and streamlining of all clusters corresponding to the target video, thereby facilitating the identification of objects for which three-dimensional position detection is required, that is, a cluster can be made to correspond to an object more accurately.

[0071] Optionally, the value of the preset threshold may be determined based on the number and specific category of objects contained in the target scene, the total number of video frames contained in the target video, and other specific circumstances.

[0072] S17, determining an initial three-dimensional position of the object corresponding to each target cluster according to the three-dimensional information of the object corresponding to each target cluster.

[0073] S19, determining the target three-dimensional position of the object corresponding to each target cluster according to the camera posture, the three-dimensional information of the object corresponding to each target cluster and the initial three-dimensional position of the object corresponding to each target cluster.

[0074] In this embodiment, when it is necessary to perform three-dimensional reconstruction on at least one object in a certain scene, the corresponding target video can be obtained, and the target video can be analyzed frame by frame to determine the three-dimensional information of the object corresponding to each video frame contained in the target video, so as to achieve the acquisition of the three-dimensional information of the object on the single-view image. Further, the three-dimensional information of the object corresponding to the multiple single-view images corresponding to the target video, that is, all the video frames contained in the target video, can be comprehensively analyzed, that is, by clustering the three-dimensional information of the object corresponding to all the video frames in the target video, and further filtering the clustering results, so that the initial three-dimensional position of the object corresponding to each target cluster can be first determined according to the three-dimensional information of the object corresponding to each filtered target cluster, and then the final target three-dimensional position of the object corresponding to each target cluster in the target video can be accurately determined according to the camera posture, the initial three-dimensional position corresponding to each target cluster and the three-dimensional information of the object. In this way, by providing a more robust performance, a wider range of applications and not limited to the need for rich texture information on the image, the purpose of more accurate object position information detection is achieved.

[0075] Furthermore, the implementation process of the above S11 may include the following contents:

[0076] Step a, determining the camera posture corresponding to each video frame in the target video.

[0077] Among them, the camera posture can refer to the Euclidean angle of the three-dimensional posture of the monocular camera relative to a known object, that is, the pitch angle, yaw angle and roll angle. In one example, when the monocular camera is a smart phone, the camera posture corresponding to each video frame can be calculated and exported with the help of the built-in Simultaneous Localization And Mapping (SLAM) algorithm of the mobile phone. In another example, if the computing power is sufficient, an offline multi-view 3D reconstruction algorithm can also be used to calculate the camera posture corresponding to each video frame. Optionally, the camera posture corresponding to each video frame in the above target video may be the same or different. Further optionally, the camera posture corresponding to each video frame in the above target video is the same.

[0078] Step b: performing object detection on each video frame in the target video to obtain an object detection result corresponding to each video frame.

[0079] Optionally, each video frame included in the target video is input into a deep neural network for object detection, and an object detection result corresponding to each video frame is output. The deep neural network for object detection may be a convolutional neural network (CNN).

[0080] Among them, the deep neural network (DNN) is also called a multi-layer neural network, which can be understood as a neural network with many hidden layers. There is no special metric for "many" here. According to the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Each layer only obtains the output vector through such a simple operation on the input vector. Since the number of DNN layers is large, the number of coefficients W (also called weight matrix) and offset vectors will also be large. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. In theory, the more parameters a model has, the higher the complexity and the greater the "capacity", which means that it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0081] In addition, CNN is a deep neural network with a convolutional structure. The convolutional neural network contains a feature extractor consisting of a convolutional layer and a subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or convolution feature plane (feature map). The convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are the convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The implicit principle is that the statistical information of a part of the image is the same as that of other parts. This means that the image information learned in a part can also be used in another part. Therefore, the same learned image information can be used for all positions on the image. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation. The convolution kernel can be initialized in the form of a matrix of random size. During the training process of the convolutional neural network, the convolution kernel can obtain reasonable weights through learning. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0082] Furthermore, the method of training the above-mentioned deep neural network for object detection involves computer vision processing, and can be specifically applied to data processing methods such as data training, machine learning, and deep learning, to perform symbolic and formalized intelligent information modeling, extraction, preprocessing, and training on the training data (such as the images of objects and the categories of objects in the embodiments of the present disclosure, etc.), and finally obtain a trained deep neural network for object detection.

[0083] Step c: obtaining the three-dimensional information of the object corresponding to each video frame according to the camera posture and object detection result corresponding to each video frame.

[0084] In this embodiment, the camera posture corresponding to each video frame in the target video can be determined frame by frame, object detection can be performed on each video frame in the target video frame by frame, and based on the camera posture corresponding to each video frame and at least one of the object detection results, the three-dimensional information of the object corresponding to each video frame can be determined to accurately achieve the acquisition of the three-dimensional information of the object on the single-view image.

[0085] Further, in this embodiment, when the object detection result corresponding to each video frame includes the physical size of each object, the posture of each object relative to the camera, and the feature vector of each object, the implementation process of step c may include the following contents:

[0086] According to the physical size of each object, the posture of each object relative to the camera and the camera posture corresponding to each video frame, each object is projected into the world coordinate system to obtain the three-dimensional bounding box corresponding to each object; according to the feature vector of each object, a search is performed in the preset object model library to obtain the three-dimensional model corresponding to each object; the three-dimensional bounding box and the three-dimensional model corresponding to each object in each video frame are used as the three-dimensional information of the object corresponding to each video frame.

[0087] It can be understood that the three-dimensional information of the object corresponding to each video frame includes but is not limited to the three-dimensional bounding box and three-dimensional model corresponding to each object. Among them, the three-dimensional bounding box corresponding to each object can be a BBOX (Bounding Box) bounding box. The bounding box is an algorithm for solving the optimal bounding space of a discrete point set, which can be used to describe the 3D posture of an object. The basic idea is to use a geometric body (called a bounding box) with a slightly larger volume and simpler characteristics to approximately replace a complex geometric object (i.e., an object); the three-dimensional bounding box can be the smallest hexahedron, sphere, or cuboid containing the corresponding object, such as Figure 3 and Figure 4 As shown, the three-dimensional bounding box is specifically a cuboid. The three-dimensional model corresponding to each object can be used to describe the prior information of the geometric shape of the object.

[0088] In this embodiment, by combining the physical size of each object in each video frame, the posture of the object relative to the camera, and the camera posture corresponding to each video frame, a three-dimensional bounding box that can accurately describe the three-dimensional posture of each object in each video frame can be obtained. Among them, the object size of each object may include measurable dimensions such as the length, width, and height of the object. The posture of each object relative to the camera may include the three-dimensional posture Euclidean angle of each object relative to the monocular camera, that is, the pitch angle, yaw angle, and roll angle.

[0089] In addition, the feature vector reflecting the category information of the object can be used to search from an existing common object model library (i.e., a preset object model library, such as a CAD model library based on computer-aided design) to retrieve the 3D model corresponding to each object. Figure 5 After searching the objects in the search, the three-dimensional models corresponding to the chairs and sofas are retrieved as follows Figure 6 As shown. In this way, by making full use of the category information of the object to obtain the prior information of the geometric shape of the object, it is beneficial to more accurately realize the three-dimensional reconstruction of the object, thereby realizing more accurate three-dimensional position detection of the object.

[0090] Among them, each element in the feature vector of an object can represent the common features of the category to which the object belongs. For example, the feature vector of a chair is (A, B, C), where A is used to characterize the features of the shape of the chair, B is used to characterize the features of the size of the chair, C is used to characterize the features of the basic structure of the chair, and so on.

[0091] Furthermore, in this embodiment, the implementation process of the above S17 may include the following contents:

[0092] According to the size of the 3D bounding box in each target cluster and the posture of the object relative to the camera, the initial 3D position of the object corresponding to each target cluster is determined.

[0093] In this embodiment, the specific size information of the measurable three-dimensional bounding box within each target cluster can be used to estimate the size of the objects corresponding to each target cluster, and further combined with the position and posture information of each object relative to the camera to achieve an accurate trajectory of the initial three-dimensional position of each object.

[0094] Further, in this embodiment, the step of determining the initial three-dimensional position of the object corresponding to each target cluster according to the size of the corresponding three-dimensional bounding box in each target cluster and the posture of the object relative to the camera may include the following steps:

[0095] The average value of the sizes of all three-dimensional bounding boxes in each target cluster and the average value of the postures of all objects in each target cluster relative to the camera are projected into the world coordinate system to obtain the initial three-dimensional position of the object corresponding to each target cluster.

[0096] In this embodiment, the size of the three-dimensional bounding box in the cluster and the posture of the object relative to the camera can be averaged first, and then the average result can be projected into the world coordinate system to complete the initial estimation of the three-dimensional position of each object in the target video. Furthermore, all video frames corresponding to the object reconstructed based on each cluster can also be determined. It should be further noted that in other embodiments of the present disclosure, the three-dimensional position of the object can also be detected by projecting into other types of coordinate systems.

[0097] Further, in this embodiment, when the object detection result corresponding to each video frame further includes the first segmentation mask of each object, the implementation process of S19 may include the following contents:

[0098] Step d: According to the camera posture and the initial three-dimensional position of the object corresponding to each target cluster, the three-dimensional model of the object corresponding to each target cluster is projected onto the video frame corresponding to each target cluster to obtain a second segmentation mask of the object corresponding to each target cluster.

[0099] Step e: determining the target three-dimensional position of the object corresponding to each target cluster according to the first segmentation mask of each object, the second segmentation mask of the object corresponding to each target cluster, and the intersection-over-union ratio of the three-dimensional bounding box corresponding to each target cluster.

[0100] It can be understood that by fusing the three-dimensional object information corresponding to all video frames in the target video, the erroneous information when detecting objects based on single-view video frames can be filtered out, that is, by cross-validating the relative position relationship between multiple viewpoints, the noise in the single-view object recognition information can be eliminated, thereby improving the accuracy of the estimation result of the three-dimensional position of the object.

[0101] The first segmentation mask of each object contained in each video frame can be a segmentation mask of each object on the corresponding two-dimensional plane (2D) image to reflect the pixels belonging to each object in the video frame image. An example of the segmentation effect is as follows: Figure 7 As shown. In this way, considering that the accurate projection of the object's 3D position on the 2D image should be consistent with the segmentation mask of the object detection and the 3D objects should not overlap with each other, more accurate object position information can be obtained by jointly optimizing the consistency between the projection of the object on the multi-view image and the single-view object segmentation.

[0102] Furthermore, in this embodiment, the implementation process of the above step e may include the following contents:

[0103] According to the first segmentation mask of each object, the second segmentation mask of the object corresponding to each target cluster, and the intersection-and-union ratio of the three-dimensional bounding box corresponding to each target cluster, the objective function corresponding to all target clusters is determined, and the objective function is used to characterize the object position and posture; the objective function is minimized to obtain a result of the minimization processing, wherein the result of the minimization processing corresponds to the target three-dimensional position of the object corresponding to each target cluster.

[0104] It can be understood that by minimizing the objective functions corresponding to all target clusters corresponding to each object contained in the target video, better or optimal processing of the three-dimensional position of the object corresponding to each target cluster is achieved, and the final target three-dimensional position of each object is obtained.

[0105] Furthermore, in this embodiment, the objective function is: Loss ,in, Wherein, N represents the total number of the at least one object, M represents the total number of all video frames in the target video, Represents the first segmentation mask of the i-th object in the j-th frame video, represents the second segmentation mask of the i-th object in the j-th frame video, IOU represents intersection-over-union ratio, Represents the collision volume between objects; i is 1,…,N, j is 1,…,M.

[0106] From the above, we can see that by minimizing the above objective function -log(Loss), we can solve the optimal three-dimensional position of each object in the target video {x_optim1,_optim2,...._optim N That is to say, by minimizing the objective function, that is, by adjusting the three-dimensional position of the object, the consistency between the projection of the three-dimensional position of the object on the 2D image and the segmentation mask of the object detection can be increased, so that a more accurate three-dimensional position can be obtained.

[0107] Reference Figure 2 , is a flow chart of a method for detecting a three-dimensional position of an object according to another exemplary embodiment. The method for detecting a three-dimensional position of an object may include the following steps:

[0108] S21, calculating the camera posture corresponding to each video frame in the captured video.

[0109] Optionally, a video is shot by a monocular camera, and the video may include common objects such as tables, chairs, sofas, cups, vases, etc. in a certain indoor scene. In one example, when a video is shot by a mobile device such as a mobile phone, the camera posture corresponding to each video frame in the video can usually be calculated and exported with the help of the built-in SLAM algorithm in the mobile phone; in addition, if the computing power is sufficient, it is also possible to consider using an offline multi-view 3D reconstruction algorithm to calculate the above camera posture.

[0110] S23, performing object detection on the video frame by frame to obtain object segmentation information, a three-dimensional bounding box, and a similar three-dimensional model.

[0111] Optionally, each video frame contained in the video is input into a deep neural network for object detection frame by frame, and the deep neural network outputs the detected object segmentation information, that is, the segmentation mask on the 2D image corresponding to each object, so as to reflect the pixels belonging to each object in the video frame image. An example of the segmentation effect is as follows Figure 7 In addition, by performing object detection on each frame of the video, the physical size of the object and its posture relative to the camera can be regressed. Further combined with the camera posture in S21 above, each object can be projected into the world coordinate system to obtain the 3D BBOX of each object, as shown in FIG. Figure 3 and Figure 4 As shown in , the three-dimensional bounding box is specifically a cuboid. In addition, by performing object detection on the video frame by frame, the deep neural network can also output the feature vector of the object at the same time, so as to retrieve the most similar three-dimensional model corresponding to each object from the CAD model library, such as Figure 5 After searching for objects in the search, the chairs and sofas corresponding to the searched objects are as follows Figure 6 The three-dimensional model shown.

[0112] S25, after fusing the information corresponding to the multiple video frames in the video package and filtering out the error information in the single-frame object detection, the initial estimation of the three-dimensional position of each object in the video is completed.

[0113] Optionally, the information corresponding to each video frame may at least include the 3D BBOX obtained in the above steps, the posture of the object relative to the camera, etc. Further, the IOU between different 3D BBOXs is used as a distance metric, the 3D BBOX obtained by object detection in each video frame is clustered, and the number of observations contained in each cluster (i.e., the number of video frames) is counted, and then the clusters with too small number of observations are filtered out to achieve streamlined screening. Furthermore, the size of the 3D BBOX corresponding to each cluster after streamlined screening and the posture of the object relative to the camera can be averaged respectively, and the average result can be projected into the world coordinate system to obtain an initial estimate of the three-dimensional position of each object in the scene; at the same time, after cluster streamlined screening, the video frame corresponding to each reconstructed object can also be obtained.

[0114] S27, optimizing the three-dimensional position of each object according to the initial estimation result of the three-dimensional position of each object in the video, the object segmentation information corresponding to each video frame, and the camera posture.

[0115] Optionally, according to the initial three-dimensional position estimation result {x1, x2, .... x N} and camera posture, the 3D model of the object can be projected onto its observation perspective (i.e., the video frame corresponding to the cluster corresponding to the object) to obtain the segmentation mask (represents the projection of the i-th object on the j-th frame of the video). Then construct the objective function -log Loss By minimizing the above objective function -log(Loss), the 3D position of the object can be adjusted, which can increase the consistency between the projection of the 3D position of the object on the 2D image and the segmentation mask of the object detection, thereby obtaining a more accurate 3D position.

[0116] in, Wherein, N represents the total number of the at least one object, M represents the total number of all video frames in the target video, represents the segmentation mask of the ith object on the 2D image of the jth frame video, IOU stands for intersection-over-union, Represents the collision volume between objects; i is 1,…,N, j is 1,…,M.

[0117] This embodiment uses a neural network to retrieve object instances and their position and posture relative to the camera on a single-view image, and eliminates erroneous information in single-view object detection through multi-view information filtering to improve the robustness of the algorithm. Then, cross-validation is performed through the relative position relationship between multiple views, which can eliminate noise in single-view object recognition information, and more accurate object position information is obtained by jointly optimizing the consistency of the object projection on the multi-view image and the single-view object segmentation.

[0118] Embodiment 2

[0119] Figure 8 FIG. 3 is a block diagram of an object three-dimensional position detection device 300 provided by an exemplary embodiment, and the object three-dimensional position detection device 300 is used to implement the object three-dimensional position detection method in the aforementioned embodiment 1. Figure 8 The object three-dimensional position detection device 300 includes a first determination module 31, a clustering module 33, a screening module 35, a second determination module 37 and a third determination module 39.

[0120] Among them, the first determination module 31 is configured to determine the three-dimensional object information corresponding to each video frame in the target video, and the target video contains at least one object; the clustering module 33 is configured to cluster the three-dimensional object information corresponding to all video frames in the target video; the screening module 35 is configured to screen all clusters corresponding to the target video according to the video frames associated with each cluster, and obtain the target cluster corresponding to the target video; the second determination module 37 is configured to determine the initial three-dimensional position of the object corresponding to each target cluster according to the three-dimensional object information corresponding to each target cluster; the third determination module 39 is configured to determine the target three-dimensional position of the object corresponding to each target cluster according to the camera posture, the three-dimensional object information corresponding to each target cluster and the initial three-dimensional position of the object corresponding to each target cluster.

[0121] In this embodiment, when it is necessary to perform three-dimensional reconstruction on at least one object in a certain scene, the corresponding target video can be obtained, and the target video can be analyzed frame by frame to determine the three-dimensional information of the object corresponding to each video frame contained in the target video, so as to achieve the acquisition of the three-dimensional information of the object on the single-view image. Further, the three-dimensional information of the object corresponding to the multiple single-view images corresponding to the target video, that is, all the video frames contained in the target video, can be comprehensively analyzed, that is, by clustering the three-dimensional information of the object corresponding to all the video frames in the target video, and further filtering the clustering results, so that the initial three-dimensional position of the object corresponding to each target cluster can be first determined according to the three-dimensional information of the object corresponding to each filtered target cluster, and then the final target three-dimensional position of the object corresponding to each target cluster in the target video can be accurately determined according to the camera posture, the initial three-dimensional position corresponding to each target cluster and the three-dimensional information of the object. In this way, by providing a more robust performance, a wider range of applications and not limited to the need for rich texture information on the image, the purpose of more accurate object position information detection is achieved.

[0122] As a possible implementation, the first determination module 31 includes: a determination unit, a detection unit and a processing unit;

[0123] Among them, the determination unit is configured to determine the camera posture corresponding to each video frame in the target video; the detection unit is configured to perform object detection on each video frame in the target video to obtain the object detection result corresponding to each video frame; the processing unit is configured to obtain the three-dimensional object information corresponding to each video frame based on the camera posture and object detection result corresponding to each video frame.

[0124] In this embodiment, the camera posture corresponding to each video frame in the target video can be determined frame by frame, object detection can be performed on each video frame in the target video frame by frame, and based on the camera posture corresponding to each video frame and at least one of the object detection results, the three-dimensional information of the object corresponding to each video frame can be determined to accurately achieve the acquisition of the three-dimensional information of the object on the single-view image.

[0125] Further, as a possible implementation manner, the processing unit, when the object detection result corresponding to each video frame includes the physical size of each object, the posture of each object relative to the camera, and the feature vector of each object, is configured to specifically perform:

[0126] According to the physical size of each object, the posture of each object relative to the camera and the camera posture corresponding to each video frame, each object is projected into the world coordinate system to obtain the three-dimensional bounding box corresponding to each object; according to the feature vector of each object, a search is performed in the preset object model library to obtain the three-dimensional model corresponding to each object; the three-dimensional bounding box and the three-dimensional model corresponding to each object in each video frame are used as the three-dimensional information of the object corresponding to each video frame.

[0127] In this embodiment, by combining the physical size of each object in each video frame, the posture of the object relative to the camera, and the camera posture corresponding to each video frame, a three-dimensional bounding box that can accurately describe the three-dimensional posture of each object in each video frame can be obtained. Among them, the object size of each object may include measurable dimensions such as the length, width, and height of the object. The posture of each object relative to the camera may include the three-dimensional posture Euclidean angle of each object relative to the monocular camera, that is, the pitch angle, yaw angle, and roll angle.

[0128] In addition, the feature vector that reflects the category information of the object can be used to search from the existing common object model library to retrieve the 3D model corresponding to each object. In this way, by making full use of the category information of the object to obtain the prior information of the geometric shape of the object, it is conducive to more accurate 3D reconstruction of the object, thereby achieving more accurate 3D position detection of the object.

[0129] As a possible implementation, the second determining module 37 is configured to specifically execute:

[0130] According to the size of the 3D bounding box in each target cluster and the posture of the object relative to the camera, the initial 3D position of the object corresponding to each target cluster is determined.

[0131] In this embodiment, the specific size information of the measurable three-dimensional bounding box within each target cluster can be used to estimate the size of the objects corresponding to each target cluster, and further combined with the position and posture information of each object relative to the camera to achieve an accurate trajectory of the initial three-dimensional position of each object.

[0132] Further, as a possible implementation, the second determining module 37 is configured to specifically execute:

[0133] The average value of the sizes of all three-dimensional bounding boxes in each target cluster and the average value of the postures of all objects in each target cluster relative to the camera are projected into the world coordinate system to obtain the initial three-dimensional position of the object corresponding to each target cluster.

[0134] In this embodiment, the size of the three-dimensional bounding box in the cluster and the posture of the object relative to the camera can be averaged first, and then the average result can be projected into the world coordinate system to complete the initial estimation of the three-dimensional position of each object in the target video. Furthermore, all video frames corresponding to the object reconstructed based on each cluster can also be determined.

[0135] As a possible implementation manner, the third determination module 39, when the object detection result corresponding to each video frame also includes the first segmentation mask of each object, is configured to specifically perform:

[0136] According to the camera posture and the initial three-dimensional position of the object corresponding to each target cluster, the three-dimensional model of the object corresponding to each target cluster is projected onto the video frame corresponding to each target cluster to obtain the second segmentation mask of the object corresponding to each target cluster; according to the first segmentation mask of each object, the second segmentation mask of the object corresponding to each target cluster, and the intersection and union ratio of the three-dimensional bounding box corresponding to each target cluster, the target three-dimensional position of the object corresponding to each target cluster is determined.

[0137] In this embodiment, by fusing the three-dimensional object information corresponding to all video frames in the target video, the erroneous information when detecting objects based on single-view video frames is filtered out, that is, cross-validation is performed through the relative position relationship between multiple viewpoints, which can eliminate the noise in the single-view object recognition information, thereby improving the accuracy of the estimation result of the three-dimensional position of the object.

[0138] Further, as a possible implementation, the third determining module 39 is configured to specifically execute:

[0139] According to the first segmentation mask of each object, the second segmentation mask of the object corresponding to each target cluster, and the intersection-and-union ratio of the three-dimensional bounding box corresponding to each target cluster, the objective function corresponding to all target clusters is determined, and the objective function is used to characterize the object position and posture; the objective function is minimized to obtain a result of the minimization processing, wherein the result of the minimization processing corresponds to the target three-dimensional position of the object corresponding to each target cluster.

[0140] In this embodiment, by minimizing the objective functions corresponding to all target clusters corresponding to each object contained in the target video, better or optimal processing of the three-dimensional position of the object corresponding to each target cluster is achieved, and the final target three-dimensional position of each object is obtained.

[0141] Furthermore, as a possible implementation method, the above objective function is: Loss ,in, Wherein, N represents the total number of the at least one object, M represents the total number of all video frames in the target video, Represents the first segmentation mask of the i-th object in the j-th frame video, represents the second segmentation mask of the i-th object in the j-th frame video, IOU represents intersection-over-union ratio, Represents the collision volume between objects; i is 1,…,N, j is 1,…,M.

[0142] In this embodiment, by minimizing the objective function, that is, by adjusting the three-dimensional position of the object, the consistency between the projection of the three-dimensional position of the object on the 2D image and the segmentation mask of the object detection can be increased, so that a more accurate three-dimensional position can be obtained.

[0143] As a possible implementation, the clustering module 33 is configured to specifically perform:

[0144] The three-dimensional bounding boxes corresponding to all the video frames in the target video are clustered according to the intersection-and-union ratio between every two three-dimensional bounding boxes in the three-dimensional bounding boxes corresponding to all the video frames in the target video.

[0145] In this embodiment, by using the IOU between the three-dimensional bounding boxes as a distance metric, accurate clustering of the three-dimensional bounding boxes corresponding to each video frame can be achieved, thereby achieving classification of each object contained in the target video.

[0146] As a possible implementation, the screening module 35 is configured to specifically perform:

[0147] Clusters whose associated video frames have a frame number less than a preset threshold among all clusters corresponding to the target video are eliminated to obtain a target cluster corresponding to the target video.

[0148] In this embodiment, the number of observations contained in each cluster (i.e., the total number of video frames) is obtained by statistics, and clusters with too small numbers of observations are further filtered out based on the statistical results, so as to achieve screening and simplification of all clusters corresponding to the target video, thereby facilitating the identification of objects that need to be detected in three-dimensional positions, that is, a cluster can correspond to an object more accurately.

[0149] Since the modules, units or sub-units involved in the above-mentioned object three-dimensional position detection device 300 have the same or corresponding technical features as the object three-dimensional position detection method in the aforementioned embodiment 1, the detailed description of each module, unit or sub-unit in this embodiment can refer to the description of the object three-dimensional position detection method in the aforementioned embodiment 1, and this embodiment will not be repeated here. For example, the detailed description of the first determination module 31 can refer to the detailed description of S11 in the embodiment 1, the detailed description of the clustering module 33 can refer to the detailed description of S13 in the embodiment 1, the detailed description of the screening module 35 can refer to the detailed description of S15 in the embodiment 1, the detailed description of the second determination module 37 can refer to the detailed description of S17 in the embodiment 1, and the detailed description of the third determination module 39 can refer to the detailed description of S19 in the embodiment 1, etc.

[0150] Embodiment 3

[0151] Fig. 9 is a block diagram of an electronic device 400 according to an exemplary embodiment. The electronic device 400 can execute, but is not limited to, the object three-dimensional position detection method provided in the embodiment of the present disclosure. It should be noted that since the object three-dimensional position detection method executed by the electronic device 400 in this embodiment has the same or corresponding technical features as the object three-dimensional position detection method in the aforementioned embodiment 1, the detailed description of the object three-dimensional position detection method in this embodiment can refer to the description of the object three-dimensional position detection method in the aforementioned embodiment 1, and this embodiment will not be repeated here.

[0152] Further, in a possible implementation, the electronic device 400 may include, but is not limited to, Fig. 9 The processor 41 and the memory 43 are shown. The processor 41 and the memory 43 are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines.

[0153] The memory 43 is used to store programs or data, such as storing executable instructions of the processor 41. The memory 43 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0154] The processor 41 is used to read / write data or programs stored in the memory and execute corresponding functions.

[0155] As a possible implementation, the electronic device 300 may further include a power supply component configured to perform power management of the electronic device 400, a wired or wireless network interface configured to connect the electronic device 400 to a network, and an input / output (I / O) interface. The electronic device 400 may operate based on an operating system stored in the memory 33, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0156] It should be understood that Fig. 9 The structure shown is only a schematic diagram of the structure of the electronic device 400. The electronic device 400 may also include Fig. 9 More or fewer components as shown, or with Fig. 9 Different configurations shown. Fig. 9 Each component shown in can be implemented by hardware, software or a combination thereof. In addition, in the embodiment of the present disclosure, the electronic device 400 can be, but is not limited to, a computer, a mobile phone, an IPad, a mobile Internet device, etc.

[0157] Embodiment 4

[0158] In an exemplary embodiment, a storage medium is also provided, and when the instructions in the storage medium are executed by the processor 41 in the electronic device 400, the electronic device 400 can be executed to implement the object three-dimensional position detection method in embodiment 1. Optionally, the storage medium can be a non-temporary storage medium, for example, the non-temporary storage medium can be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0159] It should be noted that since the object three-dimensional position detection method executed by the electronic device 400 in this embodiment has the same or corresponding technical features as the object three-dimensional position detection method in the aforementioned embodiment one, the detailed description of the object three-dimensional position detection method in this embodiment can refer to the description of the object three-dimensional position detection method in the aforementioned embodiment one, and this embodiment will not be repeated here.

[0160] Embodiment 5

[0161] In an exemplary embodiment, a computer program product is also provided, including at least one non-transitory computer-readable medium storing instructions translatable by at least one processor for executing the three-dimensional position detection method of the object in the first embodiment.

[0162] It should be noted that since the object three-dimensional position detection method executed by the electronic device 400 in this embodiment has the same or corresponding technical features as the object three-dimensional position detection method in the aforementioned embodiment one, the detailed description of the object three-dimensional position detection method in this embodiment can refer to the description of the object three-dimensional position detection method in the aforementioned embodiment one, and this embodiment will not be repeated here.

[0163] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0164] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0165] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0166] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A method for detecting a three-dimensional position of an object, characterized in that: include: Determining three-dimensional information of an object corresponding to each video frame in a target video, wherein the target video includes at least one object; Clustering the three-dimensional information of objects corresponding to all video frames in the target video; According to the video frames associated with each cluster, all clusters corresponding to the target video are screened to obtain a target cluster corresponding to the target video; Determine the initial three-dimensional position of the object corresponding to each target cluster according to the three-dimensional information of the object corresponding to each target cluster; Determine the target three-dimensional position of the object corresponding to each target cluster according to the camera posture, the three-dimensional information of the object corresponding to each target cluster and the initial three-dimensional position of the object corresponding to each target cluster; The object detection result corresponding to each video frame also includes a first segmentation mask of each object; Wherein, determining the target three-dimensional position of the object corresponding to each target cluster according to the camera posture, the three-dimensional information of the object corresponding to each target cluster and the initial three-dimensional position of the object corresponding to each target cluster includes: According to the camera posture and the initial three-dimensional position of the object corresponding to each target cluster, the three-dimensional model of the object corresponding to each target cluster is projected onto the video frame corresponding to each target cluster to obtain a second segmentation mask of the object corresponding to each target cluster; Determine the target three-dimensional position of the object corresponding to each target cluster according to the first segmentation mask of each object, the second segmentation mask of the object corresponding to each target cluster, and the intersection-over-union ratio of the three-dimensional bounding box corresponding to each target cluster; The method of determining the target three-dimensional position of the object corresponding to each target cluster according to the first segmentation mask of each object, the second segmentation mask of the object corresponding to each target cluster, and the intersection-over-union ratio of the three-dimensional bounding box corresponding to each target cluster comprises: Determine an objective function corresponding to all target clusters according to a first segmentation mask of each object, a second segmentation mask of the object corresponding to each target cluster, and an intersection-over-union ratio of a three-dimensional bounding box corresponding to each target cluster, wherein the objective function is used to characterize the position and posture of the object; The objective function is minimized to obtain a result of the minimization process, wherein the result of the minimization process corresponds to the target three-dimensional position of the object corresponding to each target cluster.

2. The method for detecting the three-dimensional position of an object according to claim 1, characterized in that: The determining of the three-dimensional information of the object corresponding to each video frame in the target video includes: Determine the camera posture corresponding to each video frame in the target video; Performing object detection on each video frame in the target video to obtain an object detection result corresponding to each video frame; According to the camera posture and object detection results corresponding to each video frame, the three-dimensional information of the object corresponding to each video frame is obtained.

3. The method for detecting the three-dimensional position of an object according to claim 2, characterized in that: The object detection results corresponding to each video frame include the physical size of each object, the posture of each object relative to the camera, and the feature vector of each object; Wherein, obtaining the three-dimensional information of the object corresponding to each video frame according to the camera posture and the object detection result corresponding to each video frame includes: According to the physical size of each object, the posture of each object relative to the camera and the camera posture corresponding to each video frame, each object is projected into the world coordinate system to obtain the three-dimensional bounding box corresponding to each object; According to the feature vector of each object, a search is performed in the preset object model library to obtain the corresponding three-dimensional model of each object; The three-dimensional bounding box and the three-dimensional model corresponding to each object in each video frame are used as the three-dimensional information of the object corresponding to each video frame.

4. The method for detecting the three-dimensional position of an object according to claim 3, characterized in that: The step of determining the initial three-dimensional position of the object corresponding to each target cluster according to the three-dimensional information of the object corresponding to each target cluster includes: According to the size of the 3D bounding box in each target cluster and the posture of the object relative to the camera, the initial 3D position of the object corresponding to each target cluster is determined.

5. The method for detecting the three-dimensional position of an object according to claim 4, characterized in that: The step of determining the initial three-dimensional position of the object corresponding to each target cluster according to the size of the three-dimensional bounding box corresponding to each target cluster and the posture of the object relative to the camera includes: The average value of the sizes of all three-dimensional bounding boxes in each target cluster and the average value of the postures of all objects in each target cluster relative to the camera are projected into the world coordinate system to obtain the initial three-dimensional position of the object corresponding to each target cluster.

6. The method for detecting the three-dimensional position of an object according to claim 1, characterized in that: The objective function is: -log Loss ,in, Wherein, N represents the total number of the at least one object, M represents the total number of all video frames in the target video, Represents the first segmentation mask of the i-th object in the j-th frame video, represents the second segmentation mask of the i-th object in the j-th frame video, IOU represents intersection-over-union ratio, Represents the collision volume between objects; i is 1,…,N, j is 1,…,M.

7. The method for detecting the three-dimensional position of an object according to claim 3, characterized in that: The clustering of the three-dimensional information of the objects corresponding to all the video frames in the target video includes: The three-dimensional bounding boxes corresponding to all the video frames in the target video are clustered according to the intersection-and-union ratio between every two three-dimensional bounding boxes in the three-dimensional bounding boxes corresponding to all the video frames in the target video.

8. The method for detecting the three-dimensional position of an object according to claim 1, characterized in that: The step of screening all clusters corresponding to the target video according to the video frames associated with each cluster to obtain a target cluster corresponding to the target video includes: Clusters whose associated video frames have a frame number less than a preset threshold among all clusters corresponding to the target video are eliminated to obtain a target cluster corresponding to the target video.

9. A device for detecting a three-dimensional position of an object, characterized in that: include: A first determination module is configured to determine three-dimensional information of an object corresponding to each video frame in a target video, wherein the target video contains at least one object; A clustering module is configured to perform clustering of three-dimensional information of objects corresponding to all video frames in the target video; A screening module is configured to screen all clusters corresponding to the target video according to the video frames associated with each cluster, so as to obtain a target cluster corresponding to the target video; A second determination module is configured to determine an initial three-dimensional position of an object corresponding to each target cluster according to three-dimensional information of the object corresponding to each target cluster; A third determination module is configured to determine a target three-dimensional position of the object corresponding to each target cluster according to the camera posture, the three-dimensional information of the object corresponding to each target cluster, and the initial three-dimensional position of the object corresponding to each target cluster; The third determination module, when the object detection result corresponding to each video frame also includes the first segmentation mask of each object, is configured to specifically perform: According to the camera posture and the initial three-dimensional position of the object corresponding to each target cluster, the three-dimensional model of the object corresponding to each target cluster is projected onto the video frame corresponding to each target cluster to obtain a second segmentation mask of the object corresponding to each target cluster; Determine the target three-dimensional position of the object corresponding to each target cluster according to the first segmentation mask of each object, the second segmentation mask of the object corresponding to each target cluster, and the intersection-over-union ratio of the three-dimensional bounding box corresponding to each target cluster; The third determining module is configured to specifically execute: Determine an objective function corresponding to all target clusters according to a first segmentation mask of each object, a second segmentation mask of the object corresponding to each target cluster, and an intersection-over-union ratio of a three-dimensional bounding box corresponding to each target cluster, wherein the objective function is used to characterize the position and posture of the object; The objective function is minimized to obtain a result of the minimization process, wherein the result of the minimization process corresponds to the target three-dimensional position of the object corresponding to each target cluster.

10. The object three-dimensional position detection device according to claim 9, characterized in that: The first determination module includes: a determination unit, a detection unit and a processing unit; wherein, The determining unit is configured to determine the camera posture corresponding to each video frame in the target video; The detection unit is configured to perform object detection on each video frame in the target video to obtain an object detection result corresponding to each video frame; The processing unit is configured to execute, according to the camera posture and object detection result corresponding to each video frame, to obtain the three-dimensional information of the object corresponding to each video frame.

11. The object three-dimensional position detection device according to claim 10, characterized in that: The processing unit, when the object detection result corresponding to each video frame includes the physical size of each object, the posture of each object relative to the camera and the feature vector of each object, is configured to specifically perform: According to the physical size of each object, the posture of each object relative to the camera and the camera posture corresponding to each video frame, each object is projected into the world coordinate system to obtain the three-dimensional bounding box corresponding to each object; According to the feature vector of each object, a search is performed in the preset object model library to obtain the corresponding three-dimensional model of each object; The three-dimensional bounding box and the three-dimensional model corresponding to each object in each video frame are used as the three-dimensional information of the object corresponding to each video frame.

12. The object three-dimensional position detection device according to claim 11, characterized in that: The second determining module is configured to specifically execute: According to the size of the 3D bounding box in each target cluster and the posture of the object relative to the camera, the initial 3D position of the object corresponding to each target cluster is determined.

13. The object three-dimensional position detection device according to claim 12, characterized in that: The second determining module is configured to specifically execute: The average value of the sizes of all three-dimensional bounding boxes in each target cluster and the average value of the postures of all objects in each target cluster relative to the camera are projected into the world coordinate system to obtain the initial three-dimensional position of the object corresponding to each target cluster.

14. The object three-dimensional position detection device according to claim 9, characterized in that: The objective function is: -log Loss ,in, Wherein, N represents the total number of the at least one object, M represents the total number of all video frames in the target video, Represents the first segmentation mask of the i-th object in the j-th frame video, represents the second segmentation mask of the i-th object in the j-th frame video, IOU represents intersection-over-union ratio, Represents the collision volume between objects; i is 1,…,N, j is 1,…,M.

15. The object three-dimensional position detection device according to claim 11, characterized in that: The clustering module is configured to specifically perform: The three-dimensional bounding boxes corresponding to all the video frames in the target video are clustered according to the intersection-and-union ratio between every two three-dimensional bounding boxes in the three-dimensional bounding boxes corresponding to all the video frames in the target video.

16. The object three-dimensional position detection device according to claim 9, characterized in that: The screening module is configured to specifically perform: Clusters whose associated video frames have a frame number less than a preset threshold among all clusters corresponding to the target video are eliminated to obtain a target cluster corresponding to the target video.

17. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the three-dimensional position detection method of an object as described in any one of claims 1 to 8.

18. A storage medium, characterized in that: When the instructions in the storage medium are executed by a processor in an electronic device, the electronic device is enabled to execute the method for detecting the three-dimensional position of an object as claimed in any one of claims 1 to 8.

Images